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TELECONFERENCING SYSTEM 

BACKGROUND OF THE INVENTION 
The present Invention relays to •elsconieretjeing systems, and 10 
5 particular to computer-based seiecooferertatig systems for enhancing coiiaboranon 

between and ainong individuals who are separated by distance and/or erne Oefetred to 
herein as "distributed collaborator^;. A system embodyu-g the tavern torfs goals can- 
replicate. Ira a desktop eJtvsronmetir and to the maximujs extent possible, the full range, 
level and intecsrty of interpersonal communication and irsfornsatiot: sharing which would 
10 occur if adl rhe p-^rticspants were together in the same room at the same tstne (referred to 
hercits as "fac«-to-faee eollabosatksn"}. 

It is well kaown so behavioral selenasts thai laterpsrscTtal 
ccnusaaicitioa revolves a large aamber of subtle and complex visual cues, referred to by- 
names like "eye conta-a" and "body langaag-e". which provide additional information 
15 over and above the spoken words and explicit gestures. These esicw; arc, for the. most 
part, ptocessed snbconsdoasiy by the participants, and often cottsroi the course of a 
meeting . 

In addrtioa to spoken words, demonstrative gestures sad behavioral 
cues. coUaborauoa oftea involves the sharing of visual infortnation - eg., printed 

20 material such as atticles, drawings, photographs, charts arid graphs, as we is as vsdeoiapes 
and computer-based aoitsaoons.. visualizations and other displays - in such a way that 
the participants can eoilecavciy and interactively examine, discuss, annotate and revise 
the biforrnauon. This cornbina-ion of spoken words, gesnrres. visual cues sad irnet^cove 
data sharing significantly enhances the effectiveness of collaboration in 4 variety of 

25 contexts, such as "brautatortning" and problem solving sessions among professionals \n a 
particular fieia. consultations between one ot snore expen.s and one or more clients, 
scastfive busmess or political negotiations, asd the like. In distributed collaboration 
settings, then, where the participants cannot be is the same place at the same time, the 
benefice effects of face-to- face collaboration will be realized o&iy to the extent that each 

30 of the re.nK.ueiy located participants can be 'recreated" at each site. 

To illustrate the dUftcaUies inherent m rspjoducit-g the bcsjeiksal effects 
of faee-to-faee collaboration in a distributed coiiaboratson envirctsrticts;, consider the ease 
of decision- tasking in the fass-nwirtg commodities trading markers, whetr. marry 


thousands of dollars or pounds of profs? {or toss) may depend on an expert ttadet makim? 
the right iicsisioii within hours, or evert onmttes, of receivntg a request from a distant 
client The expert requires immediate access to a wide range of potently relevant 
ink-rotation such 45 financial data, historical pnong ^formation, cur-cnt price quotes, 
5 ne^swire sen- ices, goveresnent policies and programs. economic forecasts, titer 
reports, etc. Much of this information car; he processed by the expert m tsoiattori. 
However, before csiakitsg 3 decision to buy or sell, he or she wiH frequently oeed to 
dtsctiss the ^formation with other experts, who may be geographically dispersed, and 
wtut the client. One or more of these other experts assy be in 3 meeting, on another call, 
SO or otherwise temporarily unavailable In this event, the expert must cotntsumcdts 
"asynchronously" - to brulge time as well as distance 

As discussed below, prior art desktop vicfcocoafeiefcong systems 
provide, at best, only 4 partial solunon to the challenges of st-xmbuted collaboration in 
real tnoe. primarily because of the-r lack of high-quaiiry video (winch is necessary for 

IS capmrrog the visual cues discussed above) and thesr ihrmed data sharing capabilities. 
Similarly, telephone arswenag machines, voice raid, fas machines and cosven-tot-sl 
electtcnic mad systems provide incomplete solutions to itse problems presented by 
deferred (asynchronous) collaboration because they are totally ^capable of 
communicating msu;-I cues, gestures, ere., and, iike convention >-Jeoconferenciss> 

30 systems, ire- generally limited * the richness of the data th&t cat: be exchanged. 

h has beets proposed to esttend traditional vidsoeonterenarsg capabilines 
from conference centers, vvhere group* of participants must assemble in the same room, 
to the desktop, whore individual participants may rema.ni m their ofike or home. Such, a 
system is disclosed, * U.S. Paseat No. 4.710,917 to Tompkins e; ah for Video 

2S Conferencing Ner^orK issued on December 1 , m~. It has also been proposed fo 

augment such video eo-tfereneing systems with limited "video mad" facilities. However, 
such dedicated vidctxonferencirtg systems t'strf extensions thereof) do not effect;*. eiy 
leverage the investment its ejsisurtg embedded information infrastructures - such as 
desktop persot-al computers a»d workstations, local area network (LAN) aod wide area 

30 aecwork (WAN) environments, bujidiog wiring, etc. - to facilitate m-cracStve sharing of 
data in she form of test, images, charts, graphs, recorded video, screen displays and the 
hke That is ; they attempt to add competing capabilities to a videocoisferertcsog system, 
rather chars adding multimedia and collaborative capabilities to the user's existing 
computer system. Thus, while snch systems may be useful m limited contexts, they do 
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not provide -he capabilities required for maxiotaily effective coSlaborsiioft, and are not 
cost-effective. 

Conversely, ^di;> at;d v«k.o eaprure and processing capabilities have 
recexstiy beers integtated tt;to desktop and portable personal computers sntl worbutjcas 
S (teretntfter getserscally referred to as - workstation "}. These capabilities have been u«cd 
primarily in desktop malranedk authoring systems for producing CD-KGM- based works. 
While such systems are capable of processing, combining. a;;d recording audio, video 
.and data locally ;i «. . at the desktop), they do so? adequately support serworked 
collaborative environments, principally due to ffie substantial bandwidth res^nre-nents tor 

50 real- time traiisrnissiort of bigh-qiwliiy. digitized audio and fuJi-niosioa video which 
preclude corxvernsoxsal LAN? from supporting more thars a few workstations Thus, 
although currently available desktop touidinedja eorr.purers frequently isjclade 
videoconferencirig and oshsr rtiuidmedia or collaborative capabilities w.thtB :Xteu 
advertised feature se? (sea. e g . A. Rsinhardt. "Video Conquers she Desktop", BYT£, 

15 September 1993, pp. 64-90), such systems have ties yet solved the. many probieros 
iartereot »• any practical tisplenjentadoa of 3 scalable collaboration sysiesv 

SUMMARY OF TEE INVENTION 

The presets iavefiaott in its various aspects is defined in the independent 
claims appended to this description. Advantageous features are set forth tn the 

20 appendant claims, 

A preferred embodiment of the present invention, is described it; derail 
below with reference it? the drawings. la ibis embodiment computer hardware, sorcwasrc 
artd eorxmmrncarions technologies are combined in novel ways to produce a multimedia 
collaboration system thai greedy facUirates distributed collaboration, in part by 

25 repressing the benefit of tace-to-face collaboration. The system righdy integrates a 

carefully selected set of tnuldmedia and collaborative capabilities, principal among which 
are desktop teleconferencing and multimedia mail. 

As used herein, desktop tekconferenckig includes reai-tsxne audio and/or 
video teleconferencing, as well as data cordetetscing Data conferencing, its mm. 

30 includes .snapshot sharing tshatit-g of "snapshots" of selected regions of the tiset's 

screen;, application sharing (shared cosiroi oi nmtuag applications), shared whiteboard 
(equivalent to shar^g a "blasfc" w«sdow} ( and associated teiepointisg and annotation 


capacities Teleconferences may be recorded anii stored for later playback, -raciudins 
both aud;o/vsdeo (A/V) sad ail data interactions. 

While, desktop telecotifereaciisg supports real-time intetac-jons. 
multimedia tnati pernios the asytscltrciicuss exchange of arbitrary multimedia docyraets?*. 
$ including 'custom-authored" messages sad previously rccotdsd ieieeonferenees. Indeed, 
st is to be understood that die mulntnedia capabilities undedyit-g sJe.jfc.top teleconferencing 
arid multimedia mall also greatly f'aciiuate the creation, viewing, and manipulation of 
hsgn-qualiry multimedia documents is. general, snchiding animations and visualisations 
that rmght be developed, forexaatpie, in she course of [afermitsoji analysis and 

50 modelling, further, these animations and visualizations may be generated for individual 
rasher than collaborative use, such that the present invention has utiliry beyond 3 
collaboration context. 

The system provides for a coiiaoosauve multimedia workstation (CMW) 
system wherein very high-quality audio and yideo capabilities can be readily 

1$ superimposed onto an enterprise' s existing computing and setwork inftastnicture, 
including wcrKsuKioas, LANs, WANs, and budding wiring 

In she. preferred embodiment, die sys;esn architecture employs separate 
real-time and asynchronous networks — die former for real-time audio and video, and 
tXi& latter for non-real -tsms audio and video, text, graphics and otfcat data, as wsi! as 

20 control signals. These networks are interoperable across ilifYetent computers (e g . 
Macintosh. Intel-based PCs. and Sua worksratioasj, operating systems (e g., Apple 
System 7, DOS/Windows, and UNIX) sad aerwork operating systetns {e.g., Novell 
Ncwars sad Sun GNC + ). In marty eases, both networks can actually share the s.ante 
cabling and wail jack connector. 

IS The system architecture ai«o accotnsno&des the situation st- wbscti the 

user's desktop computing ajid/or cotatmmieafions equipment pro% ides varymg leveis of 
media-handlmg capability. For example, a collaboration session whether real-time or 
asynchronous — may include participants whose equipment provides capabilities ranging 
from audio only (a telephone) or data only (a personal computer wfth a modems to a full 

30 complement of seal-cune, high-fidelity audio arid full -motion video, and high-speed data 
network ■facilities. 

The CMW system architecture is readily salable to very iarge 
enterprise wide network environments aceomsriOdatms: thousands of users. Further, s; is 
an open architecture that can accommodate appropriate standards. Finally, the CMW 
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iysiem iscojpotaws an inOiUive, yes powerful, uses interface, making the system ea&v to 
leans and use . 

Tae sysrem thus provides a dutritaftd mufcm-edia colkhoiwvn 
ertviroaoxent that thieves the benefits of faee-ki-face coibbocation as ceady as possible, 
leverages (~«!3ps oti to*} testing computing and network hfc-ssnscmre to she maximum 
extent possible, scabs to very large networks consisting of thousand of worlistancns. 
accormrtodstes emerging standards, sad is easy to barn and us.*. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The preferred embodiment of die Invention *dl now be descried sr. 
detail, by way of example. with reference to site drawings, in which: 

Ftgare i « 5 disgrarfijstistlc represema-ien of a rnvAnmtdiz eoshsbotatior* 

Figures 2a and 28 are tepresentaiicns of a computer screen iiiustrsttsg, 
to \he extant possible is a still image, the full-motion video and reisied user sruerfsce. 
displays which raay be generated during operation of the system of Figure ■ . 

Figure 3 sx a block sad schematic diagram of 4 "multimedia so;:al area 
jwmork" (MLAN), 

Figure 4 is a block and schemtjc diagram illustrating sow a plurality of 
geographically disperse;! MLANs of the type sttown is Figure 3 car; t?e connected via a 
wide ax»a network. 

Figure 5 $s a schetnauc dsagr&ra illustrating how collaboration sires, at 
distant lss.-.anotts Ll-LS are eonverOionaily interconnected over 3 wide area network by 
individually cotmecurtg each s-isse to every other site . 

Figsu-e € is a schema-ic diagram illustrating how cobabc-ranon sjtes a- 
distant locations LJ-LS are sEtercesatected over a wide area network usmg 3. 
t;;urci-hoppi.ng approach. 

figure 7 is a block diagram illustrating video mosaions? chesshry 
provided is the MLAN of f-jgure 3. 

Figures 8A, 8B and 8C illustrate the video window on a typical 
computer screen which may be generated during opera-Jos of Use system, and which 
contains only tbe cabee for two-parry calls <8A) and a video- tnosssc of all part-dpsms:. 
«•?• ■ ^r four-party- <8B) or eight-party <8C> conference calls. 
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Figyre 9 is 3 biock diagram iJiusuadTig audio mixing circuiay provided 
in the MIAN of Figure 3. 

Figure 10 ss ;= block diagram diusirating video cat-and-paste circuity 
provided in the MLAN of Fjgu-re 3. 
S Figure ii ss s schematic diagvatn illustrating typical operation o?" the 

video cus-atid-paste circuitry ia Figure 10. 

Figure? {consistiag of Figures J 2 A, 12S. 1.3 A, 133, UA, -48, 
iSA, I SB. 16, l?A And i?B; illustrate various examples of" h;>st- the present system 
provides video tnosaiting, video eut-arid-psstms;. artd audio mining 3; a pluraliry of 
10 distant sites for tr&amissioH over a wide atea network is order to provide, as the CMW 
of each eonfe;euce pariicjparti , video Jtnages and audio captured from the c;bec 
conference participants . 

Figures ISA sad 188 illustrate r*o different forms oi" a CMW which 
jnay be employed io the presetn system. 
0 Fkgare 19 is a schemaae diagram of a CMW add-on box containing 

integrated audio and video I/O ctrciutry. 

Figure 20 illustrates CMW Integrated with standard trmit; -tasking 
operating system and apphcauons software. 

Figure It illustrates software taoSuks which may be provided for 
20 runrting ca the MLaN Server its the MLAN of Figure 3 for cot-trolling operation of the 
AV assd Data Networks, 

Figure- Z2 ukrstratex an. enlarged example of "speed-dial" face icons of 
certain coiiaborauon participants m a Coilafcoratioe {initiator window on a epical CMW 
screen which may he geserated dnnng operation of the present system. 
23 Figsjsre 23 is a diagraxnsa&tic representation of the basic operating evesns 

occurring ixs a preferred system during initiation of a ;wo-party caii. 

Figure 14 is a block and schematic diagram iliastja-mg how physic::-! 
cortacctions are established its the MI.AN of Figure 3 for physically connecting first and 
■seeotid workstations for a two-party videoconfereoce call. 
30 Figure 2$ is a block sad schematic diagram IH-astraung how physical 

connections are established in MLANs such as illustrated in Figure 3. for a two-parry caii 
between a first CMW located at one site and a second CMW located at a remote site. 

figures 26 axsd 17 are block and schematic diagrams iliusttauti?. how 
conference bridging is provided us the .MLAN of Figure 3, 


Fjgar* 2S diagraafiwiiaiii'y iilusirates ho* a ssapsho; with annGtasierts 
may be stated in 3 pjyraliry of bitmaps dsriag data sharing. 

figure 29 is a schematic and diagrammatic illustra-iort of ihe interaction 
3«)or.§ oMHtaedb sta;{ (MMM), muMmed* cti/confoence recording (MMCR": sad 
rmilhtnedis doeujnef-f management (MrViDM) facilities. 

Figssre 30 is a scfesssaajic and diagrammatic -ilustrasior; of the- multimedia 
docatnent archtKccure employed. 

Figure MA iliasrrates a centraHsed Audio/Video Stsrs??. Server. 

Figure 31B is a schematic artd diagrsmma-tc iHustf a?;c?n 01 the 
interactions between the Audio,' Video Storage Server and the remainder of the 
System 

Figure 3JC illustrates in alternative form of the insetac-ions siivssirated 

its Figure 3 I B. 

Fipart 311) ss a schematic and diagrammatic iihistratiot- of the 
integrat»o» of M MM . MMC& a»d MMDN ftcilbfet. 

Figure 32 illustrates a get'erafwea hardware ijKpisrtietiiation of a 
scalable Audio/ V;s*co- Stotage Server. 

Figure 33 iih'sttates a higher d-r-xsghp-n versioa of the ssrvsr dilated 
in Ftgiite 32. -using SCSI -based erosspomt switching to increase die nusnl>er of possible 
suntiltanecjus file transfers. 

Fig&r* 34 illustrates she resulting auUttsaedla csOaborauon esvsronnieat 
achieved fey gw imeg ratioa of aadio/video/data teleccafetexseiag and MMCR. MMM asd 
MMDM 

Fibres 35-42 Uhmra-e a sens:-; of CNfW screen wbch may be 
generated during operati-M- of the system fat a typical seettaho involving a remote ss-x-rt 
who takes advantage of many of the features provided by the presest systems. 


DETAILED DESCRIPTION OF HIE PREFERRED 
EMBODIMENTS 

OVERALL SYSTEM ARCHITECTURE 

Referriag initially to Figure 1. illustrated therein is an overs il 
5 sJiSgramsnstic view of 3 ptet'esred tdecosferenci&g. system or multimedia coilaboravsoa 

system. As shown. eacb of s pluraiiry of "nyjUitnedia local area rsetworks" {MLANs} iO 
cojssects, vis Sine* .S, a sair&» t\ of CMW< (collaborative n j rnxed a w>A>t<n ois* 12-1 
to i'i-10 and provides sudiO'vidsc/dsta t-et^orkiag for supporting coUaborsiios atnong 
CMW users. W;\K 15 si tun* corrects multiple MLANs 10. aad typically srtdudes 
10 appropriate combinations of common carrier analog aad digital trasstnission networks. 

Multiple MLaKs ;0 ort the same physical presences, may be co&oscted via bridges/rouses 
11, as shown, to WANs aad oat aaodser. 

The system of Figure i accosnrnodates both "seal tune" de!sy-icnsi?ivc 
and jiaer-seasitive Sigtiais (e.g., rcai-sitae auoio ana v^deo signals.! and classical 
IS asynchronous cats te.g. : data control sjgsuls as *-«H as shared textual, graphics and other 
media; ectsmurucssoia asrjong multiple CMWs i2 regardless of their location. Although 
only ten CMWs i2 are illustrated in Figure I, is *tH be nude-stood that t-uoy n-ose 
coafti be: provfiie.fi As also indicated m Figure i , various other multimedia resources : ,6, 
e.g., VCRs (video cassette recorders), iasetdsscs. TV feeds, etc., ate corrected to 
20 MLANs 10 and are thereby accessible by individual CMWs 12, 

CMW 12 in Figure. I may use any of 3. variety of fcsown types of 
operating systems, such Apple System ?. UNIX, DOSfWindows and OS/2, The 
CMWs can also have different types, of window systems Specific, examples of a CMW 
12 ate described hersisaSei in cosnecuoa wtis Figaros ISA and LBB. Note that the 
25 sys-em allows for s mix of operating systems sud window sysit-.trss: seross ;n;h<-*dsiai 
CMWs. 

CMW 12 provides real-time audto/vidco/data capabilities aios-g v-th the 
usual .lata process-Jig c;rpbi;l?ta provided by Its operating sys-etn. fo: e,x:itnpie. Fig. 
?.A iHyssjates a CMW screen containing live, mll-motion video of three confere.ncs 
30 participants, while Figure 28 illustrates da-a shared and annotated by those conferees 
(lower left wiadow). CMW \2 provides for bidirectional cormrsutaca-iort, vis lines Is, 
wttfnri MLAN 10, for asidso/video signals as wet; as data signals. Aadlo.'^Kieo signals 


transmuted from a CMW 12 typically comprise a high-quality live video image and audio 
of the CMW operator. These sigtuils are obtained from a video camera and tmcrophorie 
provided at the CMW (via an add -on ursit or partially or totally integrated into the 
CMW). processed, and then tnade available to low-cost nerwork tramsmissiot- 
subsystems. 

Audio/video signals received by 3 CMW 12 from ML AN iC> may 
rypicmiv include: video images of one or more conference partse-pasts and associated 
audio, video and audio from nusitunedia mad, previously recorded andso/vsdeo from 
previous calls and conferences, and standard broadcast television {e $., CNN). Received 
v;deo sigaals arc displayed on me CMW screes ot ot; an adjaeeru monitor, and the 
accompanying audio ss reproduced by a speaker provided >n or seas -he CMW. In 
genera}, the required transducers and signai processing hardware could be mte^m-ed into 
she CMW, or be provided vu a CMW add-on mnt. as appropriate. 

U has oecfi found particularly advantageous to provide the 
above-described video a; standard NTSC-qsuirty TV performance (i.e., 30 fraraes per 
second at 640x480 pixels per frame and the equivalent of 2* bits of color per pisd) wsrij 
accompaoy-ng high-;" de&ry audio {typically berweeo 7 and 15 KHz). 

MULTIMEDIA LOCAL AREA NETWORK 
Referring next to Figure 3. iiiiis.traf.ed therein is sin MLAN SO having t«s 
CMWs (i2-i,"-i2-i0\ coupled therein via lines i3aa.t;d !3b. MLAN [Q typicaily 
extends over a. distance from around 10O metres to several kilometres (a few hundred feet 
;.o a tew raiksi, assd is usually located: witiun a building or a group of ptoxim&te 
buddings. 

Gives the current state of networking technologies, is. is useful - for siie 
«.ke c f itutris.usi.ag qualsr> At a mmnwm* <• osts) to provide separate signal paths for 
real-time audio/video and classical asynchronous data communications (including 
dtgitszed audio and video enclosures of multimedia mad messages s.hat ate free frors 
teal-time dehvery constraints). At Use moment, analog methods for eattyisg real-time 
andio/video are preferred. It; the future, digits! methods may t;e used. Eventually, 
digital audio and video signal paths may be multiplexed wub the data signal path as a 
common digital sttcam. Another alternative is to multiplex real-nine and asynchronous 
dam paths togedter using analog multiple* ing methods. For the purposes of ilinstrajioo, 
bowever. these two signal paths Ate treated as using physically separate wires. 


further. as tbs system uses analog nsswotking for audio and video, u also physically 
separates she real-time atx! asynchronous switching vehicles and, in particular, assumes 
an analog audio/video switch, in the future, a cosnnwa switching vehicle (e.g.. ATM) 
could be used. 

5 The MLaN 10 thus cats be implemented using ootsveutional technology, 

such as typjcal Data LAN fcabs 25 asd <W Switching Citcuury 36 (as used sn television 
■studios and othe; closed-circuit television networks), linked to &e CMvV's 12 via 
appropriate transceivers. and unshielded twisted pair (U?P) wiring. Note in Figure; ! that 
lines 13, wmch miereennecr. each CMW 12 wi&ia ;ts respective MLAN V 10. comprise 

10 two sets of Sines 1 3s and 1 3b. tines 13a provide bidirectional communication of 
audio-video w;d!ia MLAN 10. white lines ■ >b provide for the bidirectional 
comrnunsca-ioo of data. This separation pertnits conventional LANs so be used for data 
conxmunlcaEions and a supplemental network 10 be used for audio/video communications. 
Although this separation it; advantageous in the system illustrated, it is again ;o be 

15 understood that audio/video/data networking can also be Implemented using a Single pair 
of lines for both audio/video and data cor&rnunieatjoris v ia a very wide variety of analog 
and d;gsta.l multiplexing schemes, 

W\vte koes 1 3s and 13b may be unplerusraed In various ways, n is 
eocretmy preferred ;o use coaynonly Installed 4-palr UTP ?eiepbone wires, wherein ose 

20 pa^r is used tot incoming video *';th aceornpariVu-g audio (mono or stereo) snahjplexed 
is, wherein another pair is used for outgoing multiplexed audio/video, and wherein the 
remaining rwo parrs are used for carrying incoming and outgoing, data in ways consistent 
with existiag LANs. For example, lOBaseT Btherrset uses RJ-45 piss I.?.. 4, and 6. 
leaving pins 3. 5. ?, and 8 available- tor she rwo A/V twisted pairs. The resulting system 

25 is compaubie with cuMfer* (AT&T 25SA. EIA/TIA 568, 8P8C. lOBaseT, ISDN 1 , etc.) 
telephone wiring found coininoniy throughout telephone and LAN cable plants in most 
office buildings throug.hout the world. These OTP wires are used in a hierarchy or peer 
.wssgerciems of star topologies to create MLaN 10. described below. Note that the 
distance range of shs data wires often must match that of the video and audio. Various 

30 UTP-compa-tbU; data LAN V networks may be ased. such as Ethernet, token rssjg. FDD!, 
ATM, etc. For distances longer than the maximum distance specified by the data LAN 
protocol, dais signals cas be additionally processed for proper UTP opera-ions. 

As sbown in Fig-ire. 3, lines 1 3a from each CMW 12 are coupled ;o a 
conventional Data LAN bub 25. which facilitates the communication of data (including 

35 coiitfol Signals) among such CMWs. Lines i3b in Figure 3 are connected to A/V 


Swi?cbi5ig Cstcmtry 30. One or nwre conference bridges 35 are eoupted to A/V 
Switching Circuitry 3D and possibly {if needed) the Data LAN hub 25. via hues 35b and 
35a, respectively, for providing molts party conferencing in s particularly ad*sruageou.s 
tnanr-er, as will hereinafter be Ascribed sn desasi. A WAN gateway «0 prov-viss for 
5 &(d>recriOf53i corrsnHmicatsoB tews MLAN SO ar-d WAN J 5 in Figure I . for this 
purpose-. Dara LAN hub 25 and A/V S«dtchiag CucusLry 30 are coupled so WAN 
gateway 40 via outputs ?.5a and 30a, respectively. Other devices countxi. so the. A/V 
Swkehmg Circuitry 3D and Data LAN hub 25 to 3 dd additional features {sath as 
mnlsin^ia mad, eosferesee recording, etc } as discussed below. 

SO Control of A/V Switching Cwcuiwy 30, conference bridges. 35 and 

WAN gateway 40 in Figure 3 is provided by MLAN Serve t £0 via Sines 60b, 60c. and 
60d, respectively. In one example, MI.AN Server 60 s'rpporls the TCP/I? network 
protocol suite:. Accordingly, software processes on CMWs 12 communicate with one 
another and ML AN Server 60 vis MLAN 30 using these protocols. Other network 

IS prottxoh could also be used, audi as IPX.. The raaaiier in *h.ica software riming oa 
MLAN Server 60 controls the operation of MLAN JO will be described in deuii 
hereinafter. 

Note to Figure 3 that Data LAN bub 23. A/V Swhd-sr-g Circuity 30 
and MLAN Serve; 60 also provide respective boes 2sb, 30b, and 60e for coupling to 

20 additional multimedia resources -6 fFigatc suds as fBuHicaedia document 

manage tneat, rrtuki-sedia databases, radio/TV chatsads, ere. Data LAM ktsb 75 {via 
bridges/routers i I m Figure 1} aM A/V Swisehiag €i.retii:ry 30 additionally provide lines 
25c and 30c for coupling ro one or more other MLAN's 10 which may be in the satne 
iocaiiry (i.e., not far eaoagh away to require use of WAN technology). Where WAN:; 

75 are required, WAN gateways 40 are used ro provide highest qnahty compression methods 
and standards in a shaded resource fashion, that; ttaoitnidng costs at the workstation for a 
siivers WAN qoafiry level, as diseased below. 

The basic operation of the restdtuag collaboration system shown in 
Figares i and 3 writ next be considered Important features of the present systesi reside 

30 it- providing not only niuitl -party real-time desktop audio/video/data teleconferencing 
among geographically distributed CMWs, but also irt providiof from she same desktop 
audi;>/video/data.'test/j:rapfncs mail capabilities, as well as access to other resoatces. 
such as databases, audio and v>deo files, overview cameras, standard TV channels, e'e. 
Fig. 2B illustrates a CMW screen showing a xouHiniedia EMAIL Jiuilbo* (top left 
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window) cost&iaing refetenc.es to a number oi received messages along with a video 
enclosure (top rigin wiadow) to -fee sdected message. 

Remnsing so Figures 1 ami 3. .W Switching Circuitry- JO{*hfffer 
d;gi:d or analog} proves commaa audio/video switching fet CMWs 12, coaference 
5 bridges 35, WaN gateway 40 and txtHjtiraedia resources 16, as deterrrnaed by ML AN 
Server 60, which in ruracoatrols conference bridges 35 and WAN gateway 60. 
Similarly, asynchronous data & eorarauskated withm MLAN LG utilising cotataon data 
txmaanaicaiioas. formats where possible (e.g.. for saapshot sharing) so ri-at the system 
eaa handle such data in & coramor- manner, regardless of odgia. thereby facilitating 
10 multimedia mail sad -Sau* shards as wdl as audioAideo comranrfjcarions. 

For example, to provide multi-parry Jeiecoafsreaeing, an initiating 
CMW 12 stgtsals MLAN Server 60 via Data LAN hub 25 identifying the desired 
coafeteace panicipanrs. After desermi;uog which of these conferees will accept the call 
MLAN Server 60 eon;rois A/V Switcaiag Circuitry 30 {aad CMW software: v>a the data 
13 serwoik) to set up The required audio/video and data paths to conferees at the same 
location as s.he ij-stiatiag CMW. 

Whca one or more conferees are at distant locations, the respective 
MLAH Servers 60 of the tavotved MLAHs 10, oa a peer-to-pest basis, coatroi their 
respective A/V Switches; Circuitry 30, conference bridges 35. aad WaN gateways 40 to 
20 set up appropriate coasKt-uacaaoa paths (via WAN 15 it: Figure 1) as tequited for 

iscercftruvrctiRg rise conferees MLAN Servers 60 also eofiMftuaicate wtUa one another via 
data paths so that each ML AH i0 connsias apdated iaforraation as to the capabilities of 
all at the system CMWs 12. asd also the current locations of ail parties available tot 
{decoafetesicsag. 

25 The data ccafetetsctng cornpoaeat ol the above-described system 

supports the sharing of visual iaforttutioa at one or tnore C$4W$ (as described tn greater 
detail below). This encompasses both "saapshot shanag" (sharing "snapshots" of 
eoraptete or partial jtree&s, or of one or more selected windows) aad "application 
sharing" (sharing both rise coarroi aad display oi running appSicsuoas), When 

30 transferring images, lossless »r slightly tossy image compression cart be used to reduce 
network bandwidth reqdtenients ami user-perceived delay white matRisrniag high image 
crudity. 

in all cases, any pnrtidpaitt ca.a poiat a; or annotate the shared data, 
These associated teiepo inters and annotasio«s appear ■>« every participant's CMW screen 
35 as they are drawn ti e., effectively in real sires). Fox example, aote Figure ?.B which 


illustrates 3 typtcai CMW screen durmg a multi-party te)ecorderer-.e;rtg sessiors. wherein 
the ^recn coatams aaitotsred shared data as well as video iauges of she conferees. As 
described Li greater detail belo*-. all or portions of she audio/ video and data of the 
Kfeconfeteac* c*n be recorded at a CMW (or wfchm MIAN 10). complete wuh all the 
data iaser sciioRs . 

in the above -described ajrarigetne&t, audio/video file services can fee: 
implemented either a? the mdmduaj CMWs It or by employing a csBcrshzed- 
sudso-video storage server. This is one example of the nxsny types oi additional -.ervers 
that cas be added to the basse system oj" MLANs iO A similar approach is used for 
incorporating other multimedia services, such as eorcmeteial TV channels, ms;imns;hs 
mail, multimedia document management, multimedia cowerertce recording, visualisation 
servers, etc. (as desctibed in greater detail below). Certainly, applications that run 
seif-coatarned on a CMW cas be readily added, out the system extends this capability 
gteaiiy m the *ay -feat ML AN 10, storage sad other fuscriotts ire miplememed and 
leveraged. 

in particular, standard signal formats., network ^terraces, user interface 
messages, sad call model:; cart allow vircusliy aoy sRikimed-a resource ;c- fee smaothl> 
itttcg tiled into the system Factors facilitating such smooth integrators sticSiice: o.s a 
coKirsotJ tnechaeostn for user access across the network; (is.; a common metaphor (e.g., 
placing a call) for the user s.o initiate use ot such resource; im) the ainbty for one 
funcsiou (e.g., a multimedia coherence or multimedia database) to access and exchange 
informactott with, another function (e.g.. multimedia tnail}, and (iv) the ability so extend 
such access of oae uerworjeed faction by another rterworfced function to relatively 
complex nestings ot simpler functions (for example, record a mcifmredia conference m 
which a group ot users has accessed multimedia mail messages and transferred them ?o ;s 
multimedia daiib3.se, and tfaes send par- of the confetence recording jus; created as a new 
smiltime-dia mad message, mdulng a raafcraedia rmni editor d' necessary-;. 

A simple example of the smoo-h integration of ruactioas made possible 
by the above-described appsoaeh is that she GUI {graphical user itsierfa.ee) ac-d softwate 
used for snapshot sttantsg (described below) can also be used as an hapat/output interface 
for multimedia mad arid more geserai forms ot multimedia documents Tins cat- be 
accomplished by structuring die interprocess eoittmumcariort protocols to be uniform 
across all these applications. More complicated examples ■- specifically retitfimedji 
conference records^, multimedia mail and multimedia document mattagemeti? -■ *-d3 be 
presented txt detail feetow. 


- 14 v 

WIDE AREA NETWORK 
Next to be. described ia connection v-im Figure * is the advantageous 
manner to which -he present system proxies for res;- time audio/ video/data 
cotrnxHtrncatism among geographically dispersed M LAN's iO vis WAN IS (Figure i >, 
3 whereby comrnuniration delays, cos- and degradation oi video qualify arc significantly 
minim^ed from what would otherwise be expected 

Four MLANs iO are illustrated at locations A. B,. C and D CMWs 
12-1 so 12- 10, A/V Switching Circuitry 30. Data LAN hub 25, sad WAN gateway at 
each location correspond is show: shown, is Figures i and 3 . Each WAN gateway 40 in 
10 Figure: 4 will be seen to comprise & yourer/codee (R&C.s bank ^2 coupled to WAN 15 v-.a. 
WAN swjtchirsg mid-jplexe.r <*4. The coutsr is used Tor dai.a iraercorinectloa and the 
codec is used for audio/video latere osmection {for multimedia mail and document 
transmission, as *eli as videoconferencing). Codecs from mukiple vendors, or 
supporting vancus compression algorithms may be cssnioysd As shovra, die roster :ixi& 
SS codec are combined with d;e switching multiplexer to foxrn a single ixt-egxatsol u;ji; 
Typicaiiy, WAN IS is comprised of Ti or ISDN 
coreuxtos-cainer-provided digivai links (.switched or dedkatsd). in which ease WAN 
5»iic!uag taultipiexets 44 are of the appropmt* type (Ti, ISDN, fractional Ti . T3. 
-wiiehed 56 Kbps, etc.). Nose that the WAN switching multiplexer 44 typically creates 
20 ssibcbaxuiels whose baxid^xdfl; is a multiple of 64 Kbps {i.e., 256 Kbps, 3S4, 768, etc } 
ajfioag the Ti, T3 or ISDN carriers, Ir-vcrsc muitipiexers raay be required when using 
56 Kbps dedicated or switched services horn these carriers. 

in trse MLA.N 30 to WAN ! 5 dlxectioa. rouset, 'codec batik 42 in Figure 
4 provides coisveatic-aai" analog-!0-<i;gitaI conversion and compression of audio/video 
25 sisals received from A/V Swsidusg Circuitry 30 for ttarcranKtMi to WAN iS vj* WAN 
switching multiplexer 44, along with transmission atsd routing o;' data signals received 
from Data LAN hub 25. la the WAN 15 to MLAN ID direction, each f outer/codec 
bank -'2 ia Figure 4 provides digii&t-jo-anaiog conversion sad decompression of 
Audio/ video digital signals received from WAN 15 via WAN sivitcrkng multiplexer 44 
30 for transmission to A/V Switching Circuitry 30. along wish the transmission to Data 
.LAN bub 25 of data signals received from WAN (5 

The sysseta also ptovides optimal routes for audio/video signals through 
she WAN, For example, irs Figure 4, locatscrt A can take either a direct route to location 
D vis path 47, or a rv^o-hop rouse through location C via paths 48 and 49. if the direct 
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47 siaking ioca?o» A and location D is unavailable, the maitspaih route via |©c*>o» 
C and paths \uxd 4S> could be used. 

in a more ccsnptcx. network, several tnubs-fcop routes are typically 
available, in which case she touting syssetn handles die decision malasg. which for 
example cars be based 0=3. network loading considerations. Note the resulting jsvodevei 
network hierarchy : 3 MLAN 10 to MI..A.N 10 (Le.. site--o-iii.es service coc-neebns? 
codecs with one another only as connection endpoints. 

The cost savlags nude possible by provides the above-described 
muio-hop cspabiliry (with sriTerasediate codec bypassing) are very Significant as wiH 
become evident by soting the. examples of Figures 5 and 6, Figure 5 shows that using 
the coavenibR&i 'roily eonaec;ed mesh" iocarion-todoctuion approach, twenty-eight 
WAN Hnics are required for intcrcstinecimg the eight locations Li xo LB On the other 
hand, usiag the above n-ah;-J*op capabilities, only aine WAN Sinks are reqnhed. as 
shown is Figure 6, As the number of iocatlotss increase, che difference in cos; becomes 
even greater, F»r example, tor 100 locations, the conventional approach would require 
abau; 5,000 WAN Hnki. whsk the multi-hop approach of the present systctn would 
rypiCiilfy require 300 or fewer {possibly coas ide tab \y fewer) WAN Ihtks, Although 
specific WAN jinks tot the mulii-hop approach would require higher bandwidth ;o carry 
the additional traffic, -he cost involved is very much smaller as compared to the cost :or 
the very much larger number of WAN imk.s required by the conventional approach. 

As the etidpoints of a wide -ares call, the WAN switching multiplexer 
routes sudio/vide<5 signals directly from the WAN network Isterfaee through an available 
codec to M LAN 10 arsd vice versa. At intermediate hops in the network, however, video 
signals are routed from one nerwork interface on the WAN switching multiplexer to 
another network interface. Although A/V Switching Citcuitry 30 could be used for This 
pui-pose, the preferred syssem provides switching functionality inside the WAN swrtching 
multiplexer 8y doing so, it avoids having to route asjdi«/v 5 d«o signals through codecs to 
the at-aton swi-chirsg circuit:?, thereby avoiding addniorud codec delays at the 
iatensediate locations. 

A product eispabte of pet forming the basic swuchisg functions described 
above for WAN switching multiplexer -14 is available from Teieos Corporation, 
Satootown. New Jersey (U.S.A.). This product is not krsown to have been used for 
providing sudto-'video multi-hopping and dynamic switching araong variotis WAN hnks 
as described above 
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hi atktmoB so sue above-described niseis-hop approach, the preset 
system provides 3 particularly advantageous way of sriirarmzlag delay, cost and 
cesradauon of video quality in 3 fault? party video seleconference involving 
geographically dispersed sites, while, still delivering ruil comet ence views of at! 
5 participants. Nonnaliy, in order tor the CMWs at ail sites so be provided wtrii hve 
audi->'yideo of every participant m a teleconference simultaneously, each she. has to 
allocate nn router/codec bank 42 ia Figure 4} a separate codec for cues participant, as 
well as a like number of WAN trunks (via WAN switching multiples 44 m Figure 4), 

As will next be described, however, the sys-ets; illustrated 
1.0 advaiii&geonsiy pennies each wide ares audso/video teleconference to axe only one codec 
at eatis she, arid a mininium number of WAN digital minks. Basscaily, the system 
illustrated achieves this most issportarts result by employing "distributed" video 
jncsajcisg via a vsdeo "cut-and-paste" technology along with distributed audio mining 

DISTRIBUTED VIDEO MOSAICKING 

15 Figitre "? illustrates a preferred *sy of providsag vsdeo mosaicking m she 

MLAN of Figure 3 ~ i.e., by cornt^ng she individual analog video pictures from the 
individuals participating in a teleconference into « single analog tnosaic picture. As 
shown ifj Figure 7, analog video signals 1:2-1 to U from me participants of a 
tekax-fereaee are applied to video rriosaicing circuitry 36. which ss provided as part of 

20 conference bridge 35 tn Figure 3. These axciios; video inputs 1 H -i to i 13-si are obtained 
from the A/V Switch-ng Circuitry 30 ( Figure 3} and may include video signals from 
CMWs at one or ssore distant sites (received vs& WAN gateway 40) as well as from ots-ex 
CMWs at die local site. 

Video tuosareing Circuitry, 36. represented S;y block is capable of 

25 receiving N individual analog video picture signals (where N is a squared integer, i e. . 4, 
9.. 16\ etc.). Circuitry 36 first t educes the size of the N' input video signais by reducing 
■he resolutions of each by a factor of M (where M is the square root c-f N s'i.e, . 2 , 3, -i . 
etc), and rrtea artauging them ir. an M-by-M mosaic of N images, "ibe resulting single? 
analog mosaic 36a obtained frost! video snosaicing circuitry 36 is then transmitted to the 

30 individual CMWs for display on the screens thereof. 

As will become evident hereinafter, n may be preferable to send a 
different mosaic to dissaxst sites, i.a which case video srsosalemg circuitry 36" would 
provide at! additional tiio&nc 36b for this purpose. A typical displayed mosaic picture 
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{K~ : 4, M~2) showing three pameipants is iihtarxted in Figure 2A. A mosaic 
containing four partldpati-s is shown m Figure 8B. Is wili be appreciated thai, sines 3 
mosaic (36a ox 36b) can be transmitted as a single video picture to as other site, vis 
WAN 15 {Figures t and <>, only oas codec and digital trunk are required. Of course, U' 

5 only a single mdmduai vsdee plciate is required so be sent from a sue, it my be sen; 
d-recdy without being included in a mosaic 

Nose that for large conferences it is possible jo employ middle video 
mosaics, one for each video wisdow sapponad by the CMV>?s (see. e.g.. Figure 8C). Is 
very Urge cordeter-ces. it is Also possible to display video only ftc-ra a select locus group 

SO whose tnorubers ate selected by a dynamic "Moor control" msetiatnstn. Also note that, 
with additional mosaic hardware, it is possible to give each CMW «s own mosaic. This 
can be used in smsil conferences to raise the maximum number of participants (front M- 
so +- I - t,«.. 5, iO. if, etc.! or to give everyone in 3 large conference their own 
"focus group" view. 

13 Also note that the entire video tnossiciag approach described ituis far 

and comjriued below applies should digital video iraosmissioo be used m lieu of aaaSc^ 
transmission, particularly since bom mosaic sad video wisdom implementations use 
digital formats internally atid in current products are transformed to and from analog for 
external interfacing. In panfcuiar. nose mat aso&aicing can be done dsgiicsily wisdom 

20 decompression wirh many existing compression schemes Fctther. wi-Js a;; ail-digits; 
approach, mosaiemg cars be done ss needed direcdy on. the CMW. 

Figure 9 iihis-istes audio mixing circuitry represented by block. 38 for 
use in comnncrioa with ths vjdeo mosaic ing circuitry 36 in Figure ?. both oi which may- 
be part of conference bridges 35 is Figure 3 As shown in Figure 9. audio signals U*-i 

2S to U4-n are applied to audio taixsag or summing circuitry 38 for combination. These 
input audio signals 514-1 to ! 14~st may include audio signals from local participants as 
well ss audio surns from participants distant sites. Audio mixing cittuirry 38 provides 
a respective "saiaus-r smr, output 38-1, 38a-2« etc. for each participant. Thus, each 
participant seats every coherence panscipsat's audio except his/her cnvsi. 

3$ In the system illus-sated. sums are decomposed and formed is a 

distributed fashion, creating pamai sums a; orse site which are competed at other sites by 
appropriate signal insertion. Acectdingly, audio mixing circuitry 38 is able to provide 
one or more additional sums. such as indicated by output 38, tor sending to other sites 
having conference participants. 
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Next to be e-ynsidered is the mariner it) which video cntand-pas-e 
techniques are advantageously employed it wdl be aistierssood that, since video snowies 
and/or individual video pictures may be sent from one ot mote other sues, the oroblem 
arises as to bow these Situations are haadled. 
5 Video cut-aiid-paste circuitry 39, as illustrated its Figure 10. is ptovtded for dm purpose, 
and may also be iticorporated in she eordsrence bridges 35 hi Fignte 3. 

Referring to Figure JO, video eut-and-paste circuitry 39 receives analog 
video urputs 1:6, which may be comprised of one or more mosaics or single video 
pictures received from one or more distant sites and a mosaic or single video picture 

iO produced fey the local site. It is assumed disc the local video raosa icing circuitry 36 

{Figure 7} and the video cnt-and-paste circuity 39 have tfee capability of handuag all of 
the applied individual video pictures, or st least axe able to choose which ones are so be 
displayed based on exiting available signals. 

The video cut-and-paste circuitry .39 digisises &e incoming analog video 

15 Inputs 1 16, selective}}- rearranges the digital signals on region-fey- region basis so 

produce a single digital Mby-M mosaic, having individual pictures in selected regions, 
and then converts the resulting digits mosaic back to analog ibr-n to provide a single 
analog mosaic pict&tc 39a for sending to local participants (and othet sites where 
required! having the individual input video pictures m approptia?e regions This 

20 resulting cut-a-id-paste analog mosaic 39a wilt provide ate sarae type of display as 

illustrated ;n Figure 88. As will become evident hereinafter, it j» sometimes beneficial to 
send different c«r.-asd -paste mosaics to different sues, in which case video cist-and-paste 
circuitry 39 will provide additional cui-and-paste mosaics 39b- 3., 39b- 1. etc. for diss 
purpose. 

25 Figiue. 1 1 diagrataro&ticaily illustrates an example of h^*- video 

ast-aod-paste circuitry may operate to ptovide die cut-and-paste analog mosaic 39a As 
shown is Figure 1 1 ; four digitized individual signals 11 6a, 1 16b, Hoc derived from the 
ujput video signals a?e "pasted" into selected regions of ;s digital frame bnffer 1? to form 
a digits! 7x2 mosaic, which is coiive«ted into an output analog video mosaic 39a or 39b 

30 m Flgnxe 10- The required audio partial stuns may be provided by audio rmxmg 

circuitry Z9 m Figure. 9 in the sarae -nannes. replacing eacd cut-and-paste video operation 
with a partial sum operation. 


Having described in connection with Figures. 7- 1 I how video inosaicing, 
audio mixing, video cnt-and-pastirsg. and distributed audio mixing may be pexfotrncd. die. 


following description ■:>{ figures 12-1? win Illustrate how dtese eapabiinses ma>- 
advantageously be used in combma'ion ia tr<e context of wide-area vxieocosu'ereneing. 
Fot shese examples, the teleconference is assutned so have font participants de^igt-ared as 
A, B, C and D. in which caw 2x2 (quad) mosaic* axe ert-pioyed. it t % ?<? be understood 
S this greater siu-nbers of participants could be prodded Also, two or more 

sioiuitaneoiwiy occurnng teleconferences co-ilii also be handled, m which case addinana; 
n-osascing, eist-and-passe and aadjo mixing circuitry wouid be provided at the various 
sues along with additional WAN paths. For each example, use "A" figure sUusiratcs use 
video t-sosasertsg and cat-and-pasting provided, and the corresponding "3' figure (bavjng 
ID the same ilgure number} sHastrases the associated audio mixing provided. Note -ha- these 
figures mdicase typical delays titat might be esicoustered for each example (with a sisrgie 
"UNIT" delay ranging fross 0-4SO miSh&econds, depending upon available cotspression 
technology). 

Figures. ii'A and 128 illassrate a 2-sue example IjAvjng two participants 
15 A and S at Site *t and two participants C sad D as Sue $2. Note that ihu: example 
requires rooxudng a,«d t:ut-and-paste at bosh sites 

Figures, ; <A and I3B illustrate another 2-sttc example, bus saving three 
participants a . B and C a? Site si and one participant D as Sne Nose that iha 
example requites, mos.ascmg at both sites, bsu c-a; -aod-ps-srs eruy as She *? 
20 Figures MA and 1.4B sinistrals a 3-stte example having part- spassts A 

and B at Site #i. participant C at Site »2, asd participant D at Sise ;?3 As Sue "1 , the 
three local videos A, 8 and C are put into a mosaic wmch ss &or:u to both Sue nl and Sue 
*3. At Sue #2 and See S3, eu;-and*pas;e is used so insert die s;ngfe vxieo iC or D) at 
that sue ssuo she empty region in the imported A. B, C mosaic, a$ shown Accordingly. 
25 mosaitmg is reacted as ail three sices, and cus-and-passe u required for ostiy Site #:? and 
Site #3. 

Figures; 15a. atsd 158 diusuate another 3-sste example having participant 
A at Sue # j , participant B st Site #2, and participants. C and D as Sise »3 Note that 
moiiicajg and cat- and- paste are retired at ail sites. Sste 82 additionally has the 
30 capabdi-y to send different eut-and-paste snosaics so Sites a I ar.d Sites #3. Further note 
*«h tespets so figure 158 sbar Sire *1 creates tninus-1 audio mixes for Sste #1 assd Site 
£2, bus only provides a partial at«l«> mix ;'A&8) for Sue *3. These partial tsrixes ate 
a>t!jp!eied at Sise #3 isy rroxing ;n C's signal to compietc D';; tntx B s-C) aad D'$ 
signal to complete C's tsnx (A-s-S^D) 
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Figure i6 iilustfates a *~si;e example employing a sm topology, having 
arse participant a; each sste; thai si, participant A is at Site #l : participant S is as Si-e £2, 
participant C is r> Si?e #3, and participant D is st Site H. An audio implementation is 
no: iUusirated far Jits-s example, ■ucce s&Jidard minus- I mixing c&c< t?e performed ;k Site 
S #i, and the appropriate suras transmitted to the other sues. 

Figures I7A and i?8 illustrate a 4-site example thai aUo has ortiy one 
pirfieipam at each sire, bur uses a line topology rather than a star topology as is the 
example of Figure 16. Note snst sais cjsa-nple reqimci roosatcsng and ciK-snd-paae at si; 
sites. Also note Uiat Site #2 astl Site *3 are each required ;o transmis wo different types 
10 of cia-and-paste mosaics, 

Ths system aiso provides the capability of allowing a conference 
participant so select a close-up of a panictpam displayed on a mosaic. Thss capability is 
ptovtded whenever a full individual video picture is available as that user's site. In such 
esse, die A/¥ Switching Circuitry 30 (Figure J) switches the selected full video picture 
j 5 (Esther obtj-ised locally ot from anothet site) to she CMW that requests, the close -up. 

Next i<; tie des-cribed in cornet: with Figures ISA, iSS, 19 and 20 
are various fotxtis of a CMW, 


COLLABORATIVE MULTIMEDIA WORKSTATION HARDWARE 

Ottt form of a CMW 12 is iUuauMed tn Fig. j BA Currently 3 va>iabia 
personal computers (e.g., art Apple Macintosh or as fSM-cotnpatibU: PC, desktop or 
laptop; and workstations (e.g.. a Sun SPAHCsotion} can be adapted to work with die 
presem systers !■:> provide such features as real-time videoconferencing, data 
conferencing, tmdtiir-edia mail, est. in baseness situations, h cm be advanrageou-s to ski 
up a i;rp-op to operate with reduced functionality via cellular telephone isr&s and 
rensov*i>!c storage media <» g., CD-ROM. video tape wiUs unecode support, etc.), but 
take on full capability back in die office via a dockrng station connected so the MLAN' 
10. This reqmrcs a voice and dam modem as yet another function server artaehed to Ote 
MLAN, 

I he currently available personal computers arid workstations serve as a 
base workstation platform. The addition of certain audio and video I/O devices to the 
:;!&!)dard components of the base pktfotrn 100 (where statidatd components include trse 
display jaonitct 200, keyboard 300 and mouse or table; (or other post&ing device; 400.;. 
ill of winch connect with the base platform box through standard penphera.1 ports iOi, 
102 and 103, enables the CMW ;o generate sad receive teal-nrae audio and video 
signals. These devices mciiicie a vjsso camera SCO for capturing site user's : m;sge, 
gestures ants surroundings (particularly dse user's face and upper body), s microphone 
6(10 tot capturing the user's spoken words (and any other sounds generated at the 
CMW). a speaker "00 for presenting incoming audit} signals (such ss the spokes) words 
of another participant to a videoconiereace or audio annotations to a document), a video 
input card 130 in the base pfctform 100 for capsvtriag incoming video signals (e.g., the 
Image of another participant to a vktoeoafcnoce. or vjdeomaii), and a video display 
card 120 for displaying video and graphical output or. monitor 2.00 (wisest video Is 
typically displayed in a separate window), 

Tnsse. peripheral a«dio and video i/O devices arc readily available from 
a variety of vendors and are just tegixsatng to become standard features in (and often 
physically integrated into the monitor arid/or bass platform of) certain persona; 
compuset-s and workstanons. fe, tie aforementioned BYTE ankle ("Video 
Corners the Desktop"), which describes current models of Apple's Macintosh AV 
series personal computers and Silicon Gt*phscs' Indy workstations 

Add-on box $00 {mown in Fig, ISA and illustrated so greater detail in 
Fig. 19) integrases these asdic- and video I/O devices with additional functions (such as 
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adaptive echo cancelling and signal switching) aad interfaces with AV Network 90s. AV 
Network 90 i is she part of she MLAN (0 Thiers carries bidirectional audio and video 
signals anioag the CMWssrsd A/v Switching Circuitry >0 •••• e g. v tttihsing exsstlng UTP 
wsriag to carry aiidtc- arid video sigjisis (digital or 3*3 log}. 
S The AV Rework 90! -s separate and distinct from the Data Network 902 

portion of the MLaN iO, which carries bidirectional dsta signsis among tne CMWs 3t»i 
the Data LAN hub (e.g. , as Ethernet network that also utilizes UTP wiring with a 
sKtvs'oik iaterfacc card 110 in each CMW). Note that eacrt CMW will typ;c.:tlly be. a 
node on both tne AV and the Dm Networks. 

SO There ire several spptoacbes to implementing Add-on bos 800. In a 

typseal videocorifcrsace, video eatoeta 50s) and microphone 600 eaprurc- a.«d traasmit 
eutgoag video and audio xigaais into port* §01 and 802, respectively, of Add-on box 
800. These signals are transmitted v;a Audio/Video I/O port 80S across AV Network 
90S. inconik-g video arai audio ssgsals (from another videoeosference participant} are 

IS received across AV uetwork 901 through Audio/Video I;0 pott SOS. The v-deo steals 
ate sent out of V-OUT port $03 of CMW add-on box 800 to video lop-as eatd 5 30 of base 
platform 100. wUere they axe displayed (typically in a senate video window) on 
monitor 200 titthzk-g s.he standard base platform video display card 120. The audio 
Signals axe sen; e«i of A-OUT port 804 of CNfW add -os- box 800 and played through 

20 speaker 700 while the video signals are displayed on rcsotatot 200. The same signal How 
occurs, foe other nou-teleeonferenctag applications of audio atsd video. 

Add-oa box 8-00 can be controlled by CMW software (illustrated in Fig. 
20} executed by base platform 1 00. Control signals can be eotamuoicated between base 
platform port 104 and Add-on bo* Control port 806 (e.g.. an Centronics. SCSI 

25 Of ocher standard coa-unortlcatk-ns port). 

Many other configurations of >.cb CMW dlusmted in Fig 18a wn; 
work in the present system. For example, Add-on box. SIX! sssetf cats be implemented as 
an add-ut card to the base platform 100. Cormecuons to the audio and video I/O devices 
need not chaage, though the ceaseetion for base platform control can be Implemented 

30 internally (e.g.. via the system bits) rathet than through an externa! RS-232 or SCSI 

peripheral port. Various additional levels of integration can also be achieves as will be 
evident to those skilled in the art. for example, microphones, sneakers, video cameras 
3rid UTP transceivers cart be integrated into the base platform 100 itself, and all media 
handlist technology aad communications can be integrated onto a Single card. 
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A handset/headse; jack et&bies the use of an xtjtegrated audio I/O device 
as a;s alternate to the separate microphone and speaker. A telephone Interface could be 
integiated into add -or- box 800 as a local implementation of corapvtier-smegrated 
telephony, a "hold" (I e . audio as«S video mute) switch and/os s separate asidjc muse 
S switch eoold be. added so Add-on box 800 si' such a« inu>ietT5eos.«;c-n were deeded 
pteferabse to a software-based interface.. 

The xt-ternals of Add-on box 800 of F-g. ISA are illustrated in Pis, 19. 
Video signals generated at she CMW (e.g., capped by camera S0Q of Fig ISA) arc sen? 
to CMW add-on box 800 via V -IN port 801 . They then typically pass unaffected through 
10 loopback/AV Mats circuitry §30 via video pons 833 (isspus) ats-3 83* {output} and in;o 
A/V Transceivers 840 {via Video In port $42) wh«e they are transformed from yasdard 
video cable signals so OTP signals and sent ont via pen 845 arid Audio/Video I/O port 
805 onto AV Network 901. 

The toopbacb'AV Mute citcottry §30 can. however, be placed in 

IS various rood** URdst software coauol via Control port 80S (jjnpkswijtcd. for esampk. 
as a standard UART). If in ioopbaefc mode {e.g., for testing racotning and ousgoing 
signals at the CMW), the video cigmh would be routed bsck ouf V-OUT port $03 v» 4 
video port S3 i if In a mute mode t'e.g., muting audio, video ot both), video signals 
mtgh- , for example, be disconnected arsd no video signal would be sent otu vsdeo port 

28 §34. Loopback a-xi musing switching funaioaalisy is also provided for audso \r> a sisrtilar 
way. Nose that compter eorsLroi of loopback is very useful for remote testiag ascl 
diagnostics *-h!ie mamiai override of computer cotstrcl onrsmts is effective for adored 
privacy ft cm use of the. workstation for dearom'e spying. 

Video input (e.g., capr-ares. by she vjdeo earners at she CMW of another 

25 videoconfetence participant) is irsndled m a si-ntlar fashion. It Is received aiorsg AV 

Network §0} through Audio.' Video VO pott SOS and pon. §45 of .W Transceivers 840, 
where, it is sent out Video On; port 84 j to video pon 832 of Loopback/AV Mute circuitry 
830, which typscaBy parses such signals out video port 831 to V-OUT port 803 (for 
receipt by a video input card or other display mechanism, such as IXD display 8I0 of 

30 CMW Side Mount unit 850 ia F$g. m. to be discussed). 

Audio ir-put and ourpt-r. (eg., for playback through speaker 700 and 
capture by microphone 600 of Fig. ISA,! passes through A/V transceivers S-10 (via Audio 
hi port 8*4 and Audio Out port 843) and Loopback/AV Mute circuitry 8.30 itf-rough 
audio pons 837/838 and S3&/S3S) in a similar manner. The audio input and ourput ports 

35 of Add-on box 800 interface with suftaar;} amphfter and equahzauofj circuitry, as we!- as 
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axi active toons echo cancsi5er Si* to eliminate eeho. tninlmac feedback aaa provide 
enhanced audits performance svhen using a separate microphone and speaker. In 
particular, nse of adapisve room echo cancellers provides high-quality audio interactions 
in w»de area contereirccs. Because adaptive tootn echo cancelling requires wair.ing 
3 periods {typically involving an oineetiocabie blast of high- amplitude white noise or ■one 
sequences; for alignment with sac a acoustic etiviro«»ien!. it is preferred thai separate 
echo cancelling be -dedicated tc< each workstation rather than sharing a smaller group of 
echo cancellers across s larger group of workstations. 

Audio inputs passing through audio port S3S of Loopbscfc/AY Mute 

10 circuitry S3Q provide audio stasis to a speaker (vis standard Echo Canceller circuitry 
$14 and A -OUT port §0<i} or so a handset or headset (via VG pons §07 and §08, 
respectively, «nde? vohsme control circuitry 815 controlled by software throngh Control 
port 306}. in ail eases, incoming audio signals p&ss through power arflpUfkr cwamry 
S12 before be sag sear ous of Add-on isox S00 to the appropriate aadio-emitiing 

15 transducer. 

Outgoing audio signals generaied at the CMW (e.g., by microphone 600 
of Fig ISA or the mouthpiece of a handset or headset) entsr Add-on box S0O via A- IN' 
port 302 (for a microphone) or Handset .or Headset I/O potts. SO? and SOS, respectively 
Is all cases, outgoing audio signals pass -hreugh standard pseamptlrier (8i i) and 

20 equalisation ;'§1J>> circuitry, whereupon the desired signal is selected by standard 

"Select" switching eo.rc.unry 816 (under software control through Control port S06) and 
passed to audio port 83? of toopback/AV Mate circuitry 830, 

it is to be understood tha- A/V Transceivers 8<i0 nuvy include 
tn^ing/deinuxitrg {naaiupkmsg/ demulupkxusg} lac-lutes so at: to stable -he 

25 transmission of audio/video signals on a single pair of wires, e g., by encoding audio 
signals digitally in the vertical retrace interval of the analog video signal . 
tmpteroeBCsuon of other audio and video esxhai:£cinenr,s ; such as stereo audio arid external 
audio/vjdeo I/O pons (c.g , for recording signals generated a;, -he CMW), are also well 
within the capabilities of one skilled in the an, if stereo audio is used in telsconierenctsig 

30 (i.e , to create useful spatial metaphors for users), a second echo oar-eeller may be 
recomnteMed. 

Another configuration of the CMW. illustrated is. Fig. 1SS, ui.ihr.es a 
separate (fully self-cont&iaed) "Side Mount" approach which includes its own dedicated 
video display. Thss arraagersest is advantageous m a variety of situations, such as 
35 sns-atKes in which additional screen display area is desired (e.g.. In a laptop computer or 
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desktop system with a small monitor) or where it is impossible or ixndesirsbie so retrofit 
older, eaisrit-g or specialised desktop composers for audio/video support. Xn this 
example, video earners 500. ftsicrophene 600 sad speaker 700 at Fig. ISA are migrated 
together wjih she furictsonahty of Add-on tox 800. Side Mount 850 eianbutts s>« 
5 nccessuy of externa? corrections to these integrated audio and video I/O devices. and 
indues an LCD display SI0 for displaying the incoming v;deo signs; (which ti-us 
eliminates ate aeed for a base platform video input 
card 150). 

Giver; che proximity of Side Mount device 850 to the user, and the divest 
10 access to audlc'video VO withm that device, various additsoosj coasts 820 can be 

provided at the user's touch (si) weH wfthin the capabilities of those skilled in die anV 
Note that, with enough atUJitioiw. Side Mourn b«« 850 can become virtually a standalone 
dev;ce that does sot require * separate cojspuvtr for services <uug only audio and video. 
This also provides a way of supplementing a network of fuil-tearure workstations wich a 

!3 few iow-eost adsiisionsl "audio video intercoms'' for certain sectors of an enterprise (such 
as derieai, reception, factory door, etc }. 

A portable laptop implementation car. be made to deliver nsultM-jctha 
maii men video, audio and syacferosized aaaotaciens via CD-F.OM or an add-osx 
videotape ami with separate video, audio and rime cods tracks (a stereo videotape dayer 

20 can us.e the second audio channel for time code sigiiais). Videotapes or CD-ROMs cais 
ce created to main offices and: express .-nades. ;h;xs avoiding the need for hxgfc-bsndwldth 
networking when on me road. Cellular phone Sinks, can be used to obtain both voice and 
data communications {vis moderns). Modem-based d&u coRsn-aKicaaoas are sufficient to 
sapper? remote coruro; of mail or presentation playback, aaaotatitxis, fixe transfer and fax 

2$ features The laptop can then ;>e brought into the office and attached to a docking station 
where she available ML AN 1C -and additional functions adapted from Add-on bo*. S0G 
cars be supplied, providing tali CM W capability. 


COLLABORATIVE MOLTiMEMA WORKSTATION SOFTWARE 
CMVV software ojothties 160 are dlnst-ated generally tn Fig. 20 and 
discussed in greater detail bekw \n conjunction vs-ith the software rasaing un MIAN 
Server §0 of Fig 3. Software 160 Allows the user to niitsato and jracage (in conjunction 
5 wtrh the server software} videoconferencing, data conferencing, multimedia mail and 
other collaborative sessioas with other users across the network. 

Abo present on the CMW arc standard tnuln -tasking opsia-ing 
system/GUI st)Jrw3x» ISO (e g.. Apple Macintosh System 7. Microsoft Windows 3. { . or 
UNIX with the "X Window System" and Motif or other GUI "window manager" 
10 software? as well as other applications VKh such 3$ wors processing and spreadsheet 

programs Software modules i&i - 168 comtntir-ticate w-di operating system/GUI software 
J SO and ether applications 170 utilizing standard taction calls aod scret-apphestion 
protocols. 

The central component of the Collaborative Multimedia Workstation 

IS software ss a-e Collaboration Initiator Ibl. All collaborative fonctioas can be accessed 
through this taodtile. When the Collaboration Initiator & started, it exchanges mst;ai 
v m>* 5, ita's >» ,i'^t iaa">n w C he <U>C>o % id*o V>crk Manager s AV^M) 60 (sfio*ts 
in r;g 3} through Data Network 902. laforiaasioa is also sent from the Collaboration 
trut-atot to the AV'NM Indicating the location of the user, the types of services available 

20 on jh&t workstation fe.g.. vjdeoecaierencing, data conferencing, telephony, etc.? and 
o-hcr relevant initialisation information. 

The Collaboration Initiator presents a user msersaee that aliows the «s.cr 
to initiate coil-aburaave sessions (bosh teal-5»me and a-synchronous). Session participants 
can be selected from a graphical " rolodex' 163 that contains a scrollable list of user 

2S uames or from a lis; of quicjcKha; buttons 16.?. Quick -dial buttons show the face icons 
tot the users they represent. The scon representing the user is retrieved by the 
Collaboration Initiates from the Directory Server 66 on MLAN Server 60 when >t starts 
up. Users can dynamically acid new quick-dial buttons by dragging the corresponding 
entries ftora the graphical rolodex onto the quick-dial panel. 

30 Once the user elects ;o in*aate a collaborative session, he or she seieci.s 

one or more desired psrticipaats h>\ for example, choking on that name to sekcr site 
desired participant from die system rolodex or a personal rolodex, or by clicking os the 
qaick-dlai button or icon for that participant (see, e.g., Fig. 2As. in euher case, the user 
•hen selects the destred session type — e.g . by clicking ott a CALL burton to initiate a 
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v-,deoco»ference call, a SH A8 c. button jo ir^nste the shanng of 4 snapshot image or 
blank whiteboard, or a MAIL bncsoo >:o send snail. Aiterasriveiy. the user can 
double-click on the rokxles game or 4 iace icon to inmate the default session type — 
e.g., an audio/ video conference call 

The system also allows sessions 10 be invoked from the keyboard, it 
provides 3 graphical editor to bind combinations of participants and session types w 
certain hot keys. Presssng this sot key {possibly in cwgoKuoa win a modsfier key. 
e.g.. <Shsft> or <Ctri >} will cause the Collaboration Inmasoi to scan a session of the 
specified type svjth the given participants. 

Ottee the user selects the dessted panicipattr and session type, 
Collaboration initiator module 161 retrieves necessary addrcsstag information from 
Director)- Service 66 (see Fig. 21}. in the case of a videocosifereece ;:a.U, the 
Collaboration initiator (or, in another arrangement, VideoPbone module 169} then 
oor&tnnnfeates with the audio video network manager AVNM (as described in greater 
detail below) to set up the aec«ssaxy data structures and maaage the various states of that 
call, sod to control AA' Switching Ciresiisy 30, which selects the appropriate audio and 
video sig-tais 10 be traasanaed to/from each pajtjespant's CMW its the case of a data 
conferencing session. the Collaboration ftunator locates, via tr<e AVNM, the 
Collaboration initiator modules as the CMWs of she chosen recipients, and sends a 
message causing the Collaboration initiator modules to invoke she Snapshot Sharing 
modules 164 at each participant's CMW. Subsequent videoconferencing aad. data 
coaferencktg functionality is discussed in greases Assail below m the context of partstyiar 

As indicated previously, addiiJOiiai collaborate service's — such as 
Masi 165, Application Sharing 166, Comp^cr-integrasesi Telephony 167 and Compter 
Integrated Fax 168 - are also available ftosn the CMW by utilizing Collaboration 
Imitator module 161 to initiate the session (i.e.., to contact the participants) and to invoke 
site appropriate application necessary to maitage the collaborative session. When 
initiating asyfichroaous collaboration (e g.. mall, fax. etc.), the Collaboration Initiator 
contacts Directory Service 66 for address information (e.g., EMAIL address, fax 
number, etc.) for -he selected participants and invokes the appropriate collaboration toois 
w>th the obtained address information For real-time sess=.ons. she Coiiabotation initiator 
queries the Service Server module 6§ inside AVNM 63 for the current location of the 
specified participants. Usscg cbls location it; formation, it communicates (via the AVNM; 
with the Collaboration initiators of the other session participants to coordinate session 
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setup. As a result, the various Collaboration t-iitiascsrs will invoke nicd-ales 166. 167 or 
16S (including activating any necessary devices such as the eoruiectioa Oeoveen the 
telephone and the CMW's audio I/O pen). Further details on multimedia, mast Ate 
provided below. 

5 MLAN SERVER SOFTWARE 

Figure 2,1 diagritrnmatically illustrates software 62 comprised of' various 
modules (as discussed aOovc) provided tot rttajuag on M1.AN Server 60 (Figure Tt It is 
to be understood that additional software (nodules could also foe provided. H is also to be 
understood that, although the software Illustrated in Figure 21 oilers various significant 

10 w advantage-*, as will become evident hereinafter, differest forms and arrangements o; 
software may also be employed. The- software ca.u also be implemented is various 
sue-part running as separate processes. 

Clients <Ag-. software controlling workstations. VCRs, laserdisc, 
multimedia resources, etc.) contreuascate with the MLAN Server Software Modules 62 

15 using the TCP/FP network protocols. Get-srally. she A VNM 63 cooperates with the 
Service Server 69, Conference Bridge Manager <€BM 6* In .Figure 2\) and the WaN 
Network Manager fWNM 6S -,o Figure 2\) ;o manage communications within and among 
both MLANs iO and WANs 53 (Figures 1 and Tt. 

The A VNM additionally cooperates with A\Kho.' Video Storage Server 6"? 

70 and other multimedia services 68 in Figure 2 i to support various rvoes of collaborative 

mtetacMOHs as: described herein. CSM 64 in Figure 21 operates, as a client of the A VNM 
63 to manage conferencing by controlling the operation of conference bridges 3.5. Tins 
includes nusaageraeas of the video mosaieing circuitry 37, audio tnixitig circuitry 38 and 
cut-aod-paste circuitry 39 preferably incorporated morels. WNM 65 manages the 

25 :siiot::iticta of paths {codecs ana' tracks.) provided by WAN gateway 40 far accomplishing 
the communications to other sites called for by she A VNM, 

Audio Vidse Newark Manager 
The. A VNM 63 roar-ages A/V .Wcning Circuitry 30 in Figure 3 for 
selectively routing audio/video sigsals to and from CMWs 12. and also to and from 
30 WAN gateway 40. as called for »v clietas A«ah>Mdeo devices (e.g., CMWs 12. 

conference bridges 3S, msjitiaaeiiia resources ;6 and WAN gateway 40 in Figure 3) 
connected to A/V Switching; Circuity 30 in Figure 3, have physical connections for 
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audso m. audio cur, video its and video out. For each device or the network, the aVNM 
combines rhe-se four cortxieesions auo a port attraction, wherein each port represents an 
addressable fcidirecttoaaS audio/video chanad. Eaea device cotsntcted jo the network has 
&( ^ast one port. Different poets may share the same physical corrections; or? -he switch 
5 For example, a conference bridge may typically have fot-r ports (for 2x2 f:jo5U)cina) that 
shirs the same video-oar srorases&ori. Net all devices seed both video and audio 
connections at 3 port. For example, a TV sxiset port needs only itjcorrsirsg audio' video 
connections . 

!a responds jo client program requests, the AVNM provides coaoecs.iv-.jy 
10 between audio/video devices by connecting their potts. Connecting ports is achieved by 
swltehoig one port's physical snput connectioris to the other porfs physical ompijt 
connections (for both audio and video} and vice- versa. Client programs can specify 
w;r.c:h of -he 4 physical cots&eettoas on its ports should be switched. Thi-i allows citem 
pfqgMmt to establish ttaidifectjot-ai calls (e g., by specifying feu urriy me pc-n' ;■: input 
55 corvee dons should be swhchsd and no; the port's output coin-sections) and audio-only or 
video-only calls (by specifying audio connections otsiy ot video connections only). 

Service Server 

Before client programs car: access audio/video resources through the 
aVNM. they must register she collaborative services they provide with the Serv ice 

20 Server 69. Examples of these services indicate 'video call", "snapshot sharing". 

"cord'ererjee" and ~vsdeo file sharing" These service records are entered into me 
Service Server's service database The service, database thus keeps track of the Soca-ion 
of client prsigjasss and the types of collaborative sesssoas m which they cars participate. 
This allow;; the Collaborates tnittaior to find collaboration participants ao starter where 

25 they ate ;oca;ed The service database is replicated by ail Service Servers: Service 

Servers corrussaaicste with other Service Servers in other MLaNs throughout the system 
to exchange cheir service records. 

Clients may create a plurality of services, depending 00 the collaborative 
capabilities desired. Wiser! ctea-rng a service, a client can specify !he network resources 

30 ("e.g. pons} that wsll be used by this service. In particular, service -.ofotuiadoir is used so 
associate a user -vvsth the audioAddeo ports physically connected to she particular CM W 
into whicri the user is logged la. Clients that wast to receive requests do so by putting 
their services tn listening rrsode. If clients want to accept incoming data shares, bc-t want 
ro block incoming video calls, -hey must create tslficrent services 
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A cheru can create an exclusive service on a set of potts so prevent oih 
clletus from creating services on these pom. Tins ss useful, for example, so prevent 
multiple conference bridges from managing the same set of conference bridge ports. 

Kent to b« considered is the preferred manner m vvsuch she AVNM 6? 
(Figure 21 >. in cooperation with the Service Server 69. CBM 64 sad participating 
CM Ws ptovide for nsasag ing A/v Switchiag Circuitry '30 aad coafeteacs bridges. 35 srs 
Figure 3 during audxo.''vidco?data rekconferer-eirig. The participating CMWs may 
itidudo workstations located at both l-x.nl and remote sites. 

BASIC TWO-PARTY VIDEOCONFERENCING 

As previously described, a CMW includes a Collaboration Initiator 
software tsKiiitsie: 161, {see. Fig. 20} which is used so establish person-to-persou aad 
multiparty calls.. The corresponding collaboration initiator window advantageously 
ptovidea qufck-dtai face icons of frequently dialled persons, as illustrated, for example, 
it: Figure 2?., which is an enlarged view of typical face icons along ^ith vanous -nsfsatsr 
buttous (.described m greater deuil below in connection wits Figs 35-^2) 

Videc^onference calls can i>e initiated, for example, tircrely b> 
double-clicking on these icons. When a call is initiated, the CMW rypicaiiy provides a 
screen display that includes a live video picture of -he remote cofiferesce participant, as 
illustrated for example in Fsgura 3A This display also includes control butsons/menu 
Hems that can be used so place the re-note participant on hold, so re;.o.n;e a --ail ov> hold, 
;o add one or more partkipaxux to Use cali, to initiate data sharing aad to hang up she 
call. 

The b&sxc underlying sa;ftware.--coatro;ied operations occurring for a 
two-party coil sr/e dlagrammatica-Sly illustrated m Figure. 23 Alter logging so AVNM 6 
as indicated by ( I ) is Figure 23. a caller initiates a ca.il (e.g , by selecting a mizx fro;n d 
graphics; roiodex and clicking she call button or by double-clicking the face scon of die 
callee on she quick-dial panel). The caller's Collaboration Initiator responds by 
identifying the selected user and requesting that user's address from Directory Service 
66. as indicated by {2} us Figure 23 Dxteorory Service 66 looks up the caileeH address 
m the diteaory database, as indkated by (3) in Figure B, aad then returns if so the 
caller's Collaboration Isniiatot, as illustrate:;! by (4} m Figure 23. 

Tsse cadet's Collaboration Initiator sends a request to die AVNM to 
place a video call ?o the caller wid- tbe specified sodtess, as indicated by (S) in Figure 
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23. The AVNM queries the Service Server to fiad me service instance of type "video 
call" whose name corresponds io the caske's address. Titi? service record identifies the 
iocst-orr of the caliees Collaboration inuiator as *ei! as the ncrwotk sons that -he caiiee 
is connected to. If no service usance, is fownd for -he caiiee. the AVNM noribes the 
caller that the calies « aa logged is. if the caiiee is local the AVNM send;; a call event 
ti> s.be ealiee's CoUaboratiets inh.tcst.or, as mdicafed fey {6} in Figure 23. If she caiiee is as 
a reraote s-te, she aVN'M forranb the caii request (5) through the WAN gateway 40 tor 
aaitsBRissiOB. via WAN 15 (Figure I) to the Collaboration initiator of -he caiiee s CMW 
as. site remote sue. 

The ealiee's Coliaboraboa Isiskior tars respond to the caii evess? its a 
v;me;y of ways A user-sdeciabie sound is paerated to ansautice the incoming caii. 
The CoHa!x.-ra»oa ituu&tot can thee act ia otse of two taactes. its "Telephone Mttfn". the 
Oostaoorat-ott ijiiuaror displays aa ifiviSMien message on the CMW screen shat contains 
the t-a^te of the esiier arid buttons to accept or refuse the call The Collaboration 
iaa.sa.tcr the;; «ccept or refwse the call, depending on which bunco is pressed by the 
caiiee. Ik 'Intercom Mode", the Collaboration Initiator accepts all incoming caii* 
automatically, uaiess there is alseady another caii active ok the ealiee's CMW, in winch 
esse behaviot -evens to Tetephoce Mote. 

The csHee's Collaboration Initiator ibsii notifies the AVNM as so 
whether she caii will be ac:»p;e.d or refused, if the call is accepted, (7). the AVNM jess 
up etc accessary cotranunkattoa paths betwoc-a the caller and the caiiee reeuised so 
establish the caii. The AVNM then aotiiies ?he caller s Collaboration Initiator that the 
caii has bees established by serjdiag tt an accept event (g) If the caller and caiiee are at 
different sites, their AVNM* will eootdiaass is setting up the cotu»uakM»o paths at 
bo;.h sires, as required by the call 

The AVNM may provide for rnanagntg connections aaiorsg CMWs and 
other muimncdia resources for atsdioAtdeo/data conanaaicatiora i.ts vatious ways, is a 
manner which will next be. described. 

As has been described previously, the AVNM manages the swjtches in 
the A.-V Switchiag Ctreufary 30 is Figure 3 so provide pon-to-pott corrections ia 
response to coascchtsa requests t'rom clients-. The primary da-a structure uses by she 
AVNM for managsag these cosneetSons will be referred to as a cailhandle. whseh is 
comprises of apiaraiay of has, itsdodisg state bus. 

Each port-to-port cooaeaioa auoaged by the AVNM comprises two 
caiihaxsdles. one associated with each end of the coanscnoa. The eaJlhandte at the clteas 
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port of the connection penrnts Use ehent jo manage the cheat's end of the connection. 
The cailhassdle snode has deterrctme the current state of the caiihandle and which of a 
port s four switch connections <; video in, video out audio i;s, audio out) are invoived a; a 

.5 AVNM clients send call reqisesLs to shs? AVNM whenever they want 10 

initiate a call. As parr of a call request, me client specifier the local service in which the 
call '-viil be involved, die satss of she specific port to use for the call, identifying 
inforrsstion as ro the. calke, and site call tnode. In tespense. the AVNM crests* a 
caslhsndle or. the. caller's port 

SO All cailhandies are «eated in the "idle" aate. The AVN'M then puts the 

caller';. callhandle sn the "active' st«te The AVNM next creates a cailhaadie for the 
cailee and sends it a call event, which places she callee's callhantite ia the "ritsging" s;ate 
When the. eallee accepts the call, its eailhaadie ss placed in the "active* state, which 
results in a physical connection between the caller and the eallee Each port can have an 

15 stbitrary xtsitr.teer of c.alihandles bound to u« but typically only one of these callhandies 
can be active at the same time. 

Afkr a call has been set up. AVNM clients can send requests to the 
AVNM to changer the state of the cat), winch cars advantageous!? be accomplished by 
controlling the eailhandie states. For example, dnstof a call, a call teques; ftoro another 

20 party could arrive This arnvsl could be stalled to she usee by providing at; alert 

indication in a dialog bos; on the uses'; CMW screes. The user could refuse she call by 
clicking on a refuse buttos in the dialog box. or by clicking on a "hold" butsca on she 
active call window to put the cuitem call on hold asd allow -he incotrshig call to be 
accepted. 

15 The placing of the currency active call or hold can a£vasit.agcovi<;ly be 

accomplished by changing the caller's callhandle from the active state to a "hold'' stare, 
*htch permits the caller to answer ineommg calls or sn-tsate new calls, without teleastng 
she previous call. Since, she connection set-up to the callee will be retained a call on 
hold can conveniently be resumed by the caller clicking on a resume buaon on the active 

30 call window, which returns the corresponding cailhandic back, to the active state. 

Typically, multiple calls can be put on hold in th;s manner. As as a;d In managing cails 
that are on hold, the CMW advantageously provides a hold list display, identifying- tnese 
on hold cails and (optionally) the length of rinse tlist each party is on bold. A 
corresponding face icon could be used to identify each on-hold call. In addition, buttons 

35 could be provided In das hold display which would allow she user to send a 
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preprogrammed message so a party on hois. For example, ?h;s inexsage cotdd advt*e she 
csliee *hen the c:;US wsH be resumed, ot could state that the call is being terminated and 
w>i; be reinitiated as a laser atne. 

Reference a now directed so Figure 24 which di«gtaoM»«»tiiii>- 
illustrates how two-party calls are connected tor CMW> WS-i asd W3->, soeased a? rfcs 
san-e Ml.AN 50 As shown its figure 24, CMWs W$; and WS-2 ire co&pled to she 
local A/v Swjtchi&g Cucwtiy 30 via ports Si and §2, respectively As previously 
desenbed. when CMW WS<} calls CMW WS-2, 3 cafihandle i& crested for eaeh port If 
CMVV accepts the caii, these rwo c&iifcaadies become active and if- fescomc 
thereto, the AVNM causes the A/V Swi-chmg Circuisry JO k; set up the appropriate 
cont:ec:;rons between ports Si and S2, as jadicated by she clashed Use S3, 

Figure 25 diagrarxtrruticaily illustrates how rwe^parry calls are corrected 
for CMWs; WS- i sxid WS-2 wi-en located m differed MLANs Ida and 50b As 
illustrated 10 Figure 25. CMVV' W3-1 of MLAN I0a ;$ connected to i pots. 9!s of A/V 
Switching Circuitry 3C>3 of MLAN 10a, whvie CMW WS-2 is corrected to a port of 
the aiKho/vsdeo witchim circuit 20b of MLAN 10b. it will be asstwned that MLANs 
10* and 10b can cc-rnrouxrkate wsth each other via ports 92a at-d 92b {through respective 
WAN gateways 40a and 40b and WAN 15) A call between CMWs WS-| and WS-2 can 
then be established by AVNM 0? MLAN iOa in response to the creation of caiihandles at 
pom 91a :md 92a, seausg up appropriate coaxtections berween these pom as indicated by 
dashed Use 9Ja : at-d by AVNM of MLAN 50b. in response to eallhandies created a; 
ports 9lb and 92h, sorting rip appropriate connections between these pons as itsdiaawd by 
dashed line 92b Appropriate, paths 94« and 94b in WAN* gateways 40a at-d 40b. 
respectively arc. set up by the WAN network manager 65 (Figure 7i) m each network. 
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CONFERENCE CALLS 

Next to be described b the specific rrtaanet ta which the system 
lllnstra'ed provides for multi-party cotsfarence calls (involving tnore than ewo 
participants?. When a multi-party conference sail is initiated, the CMW provides a 
S screen that is similar ;o the screen for two- patty c:Uis. which displays a ilve video picture 
of the csilee's sxn&gc its a video window However, for mulu-parry calls, the scteen 
mdudes a video mosaic corstaimng a Uve video picture of each of the contete nee. 
participants fnx-luding rise CMW user's own picture;, as she*-;, for example, m Figaro 
SB Of course, other arrangements could show only the remote eom'ereoce participants 

iO (and not the local CMW user) ;h the conference mosaic (ot she;* a mosaic containing 

boils participants so a two-party call). In addsoon to the controls shown in Fsgvtre SB, the 
multi-parry conference screen also includes burtcrts/'meno. items that can be used to place 
individual conference particspacis on hold, to remove individual participants forai the 
conference, to adjenm die entire conference, or to pfcvtdc a "doss up" iroage n? a single 

15 individual (in place of she video mosaic; . 

Mo!r.i-oarry conferencing requires ail the. roechasnssns employed for 
2 -patty calls. In addition, it reousres die conference bridge maaaget CBM 6* (Figure. 
21) and the conference bridges 36' (Figure 3). The CBM acts as. a ciieru of the. ;WNM in 
managing the operation of die conference Bridget 36. The CBM also «ct» a server to 

20 other clients on the network. The CBM makes coafcreaasg services available by 
ttesting servsce records of type "conference" in site AVNM service database and 
associating these sendees with rhe ports on A/V Switching Circuitry 30 for ooonocttoo to 
conference bridges 36. 

The system provides two ways tor initiating a conference call The firs; 

25 *ay is to add one or mote parties to an existing two-party call. For this purpose, as 
ADD burton is provided by both the. Collaboration Initiator and die Rolodex, as 
llinstratsd irt Figures 7.A and 22 To add a new party, a user selects die patty to be 
added (by cHcfcjag ot: the. user's rnlodes name ot face Icon as described above) and clicks 
an the. ADD button to revise ths.% new party. Additional parties can be invited in a similar 

30 manr-er. The seeotxl way to initiate a conference call is to select the parries in a ssmtlar 
manner and then click on the CALL button (also provided m the Collaboration Initiator 
and Rolodex windows on the user's CMW screen). 

Another alternative possibility ss to initiate a conference eats ftont the 
beginning by clicking on a CONFERENCE/MOSAIC icon/button/menu stent on the 
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CMW screen. Th;s could smtiate a coafetetice cat! with the. call initiator as the sole 
participant 0 e , causing a confeteuee bridge so be allocated such chat a>.e caller's inwge 
also appears on his/her own screen m a video mosaic, which wHi alto mciude iraages of 
sutnccpeMly added participants). New participate could he snviseti, for example, fey 
5 selecting each new panyH face icon and Uiea clicking on the ADD b««cn. 

Next to be considered vvisi; refe.reace to Figures 2<S and 2? Is the manner 
in which conference; calls are handled. For the purposes of titss description it wij be. 
assumed that up to four parties may participate, ,m a eoafetefsce east. Esch conference 
uses foiif bridge ports 136- 136-2, 136-3 and 136-4 provided on A/V Switching 

10 Circuitry 30a. which are respectively coupled to bidirectional audio/video Imes 36- 1 , 

36-2, 36- > arid 36-4 connected to confe-tenee bridge 36, However, from dm descriptor* 
is wiil be apparent how 3 conference call may be provided for additional parties. as well 
as simultaneously occurring conference calls. 

Once the Collaboration Initiator deteraat-es that a conference 1* to be. 

IS i-attared, it queries die A VMM for a conference service. If such a service k available, 
die Collaboration Irotiator requests the assoesased CBM to allocate s coherence bridge 
The Collaborator; iniaasor then places an audio/v$<fco call so the CBM to initiate the 
conference. Whea the CBM accepts the call, the AVNM couples pott 101 of CMW 
WS-I to Suses 36-1 of conference bridge 56 by a connection i 37 pr-xlucsd xrt response ro 

20 callhandles created for port 1 Ot of WS-l aad bridge port 136- 1 , 

When the. user of WS-1. selects the appropriate face icon and clicks the 
ADD button to invite a sew participant $0 the. conference, which wih be assataeti to be 
CMW WS-3, the Collaboration h-itiator on WS- 1 sends an add request So the CBM in 
response, tbe CBM calls WS-3 via WS-3 port 101 Wl-et- CBM' initiates the call, the 

25 AVNM creates callhaadks for WS-3 port 103 and bridge port 136-2. Wises WS-3 
accepts the call, its cailhandle is made "active"', resuitiug in connection 13S being 
ptovided to connect WS-3 and lines 136-3 of conference bridge 36 Assuming CMW 
WS-l next adds CMW WS-S and then CMW WS-$. callharsdles for their respeenve pons 
and brxdge pons 1.36-3 and 136-4 are created, in turn, as described -above tot WS-I and 

30 WS-3. resulting in connections 13$ and 140 bekug provided $0 connect W$-3 and 

io conference bridge iu-es 36-3 and 36-4, respecnvely. The conferees WS- I . WS-3, 
WS-5 ;srsd W$-§ ate thus cot-pled to conference bridge lines 136-1 ; 136-2, 136-3 and 
136 -4, respectively as shows is Figure 26. 

h will be understood that die video mosaicing ctrcaiay 56 and audio 

35 mixing circuity 38 incorporated in conference bridge 36 operate as previously oesenbed. 
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50 fcxxa a resulting ibur-pitrotre mosaic (Figure SB) that Is sent to all of the conference 
participants, which ia this example are CMWs WS-i, WS-2. W3-5 and WS-3. Usees 
tnay kave a oonletence by just hanging up, which causes die AV'NM <o delest the 
assoaaied eaiihaadSes and to send a haagcp notification <e CBM. When CBM receives 
S the notification, it notifies &l\ other conference participants that the patticipanj has exited. 
This results ia a biscker-sd portion of that parttcipaas's video rnosasc linage bemg 
displayed on rhe seresa of ail retasiaiag participants. 

The manner in which the CBM and the coafeter-oe bridge 36 operas 
w&sa coafereace participants are located at different sites will be evident horn she 

50 previously described operation of the cut-aod-psste circuitry 39 (f sg«re 10) wuh the 
video mosaieing circuitry '56 (.Figure 7) atsd audio raring circuitry 38 {Figure 9V In 
such case,, each incoming single video picture or mosaic from another site is connec-ed so 
a respective ox-s of the conference bridge lines 36-1 to 36-4 vis WAN gateway <J0. 

The situation in which a rwo-psny call is convened to a conference cali 

13 svid next be consisted in connection wjth Figute 2" and ihe previously considered 

2- party eail Illustrated ia Figure 24. Convening this 2-party cah so a conference; requires 
that this two-parry call (snot; as illustrated between WS-J and W5-2 in Figure 24; be 
rerouted dynamically so as to be coupled through conference bridge 36. When the user 
of WS- i clicks on she ADD button to add a new party, (for example WS-5), Use 

20 Collaboration initiator of WS- 1 sends a redirect request to 'die AVNM which cooperates 
wi& the CBM to break the two-party connection S3 in Figure 24, and then redirect the 
ealirsaadies treated for pom Si and §3 to caiihandles created for bridge pons 136-1 arid 
1 36-2, respectively. 

As shown in Figx're 2'?. this results in prrxlnclag a coaneciiots S6 

2S berween WS-S and bridge post S36-1, and a coiiaection between aad bridge port 
136-2, thereby creasing a conference set-up beevveet- W'S-j aad WS-2. Additional 
conference participants can diets t;e adtied as described above for Use ihttratiOsu described 
above ia which die coafctetice is initiated by the user ot WS-i either selecting multiple 
participants ird-iahy or merely selecting 3 "conference" and then adding subsequent 

30 participants. 

Having described -he preferred manner in which rwo-parry tails and 
conference calls are set up, die preferred manner in which data conferencing is provided 
be-weeri CMWs next be described. 


DATA CONFERENCING 
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Data conferencing is ircpiertietned by cenain Snapshot Sharing software 
provided a? the CMW (see Figure 20). 7h;s software permits 3 "snapshot'' of a selected 
peniors of a pamdpa»Ts CMW jcrsea <s»ch as a window) to J* displayed on the CMW 
screens of othe; selected participants (whether or act those panicipares arc also savoived 
5 m 3 videocosference). Any ntsrabcr of snapshots may be shared sinniKaneously. Gnce 
d;sp-Sayed, any parfinparu can -her- teleposot on or annotate ;ho snapshot, which animated 
atitofis and results wis! appear (vsnuaiiy simultaneously} on the screens of all other 
pari:ie;pa.ms. The aaa&tatioa capabilities provided include Hnss of several different 
widths and text of severs- different sizes. Also, to facilitate participant identifk&noo. 

10 these sssotadons tji3y fee provided m a different color for each panic jpsar Any 
annotation may also be erased by say participant. Figure 2B (lower left wjp.dow) 
iiiustrares a CMW screen having a shared graph on which participants have draws' and 
typed to call anefttios so or supplement specific portions of the shared image. 

A participant -nay Initiate data conferencing with selected participants 

15 (selected attd added as described above for videoconrerefioe eaiis) by clicking w a 
SHARE burton on ite scre.es (available io the Rolodex or Coilabotanon feitiatot 
windows, shown in Figure 2A, as are CALL and ADD bisons), followed, by selection of 
Oie window -o be shared. When a participate clicks on his S.HAR.S btototu his 
Collaboration Imustor module 161 (Figure 20} queries the AV'NM to locate she 

20 Collaboration Initiators of die selected participants, resuitusg m invocation of their 

respective: Snapshot Sharing tnoddes 16*. The Snapshot Sharing software rnodntes a; 
the CMWs of each of the selected participants query jheir iocd operating system 180 to 
deterrc-use available graphic fnrrrsacs, srsd then send this ktfoxnustiots to the hikiatiag 
Snapshot Sharing module, which determines the format that wxil produce the most 

25 advantageous, display ^asiity attd performance for each selected partseipans . 

After she snapshot to be shared is displayed on all CMWs, each 
purriopant may telepoist <sn or annotate site snapshot, which action and rcsuLts are 
displayed on ;he CMW screens of ah participants. This is preferably accomplished by 
monitoring the acnoas made at ths CMW (e g., by tracking mouse movements} sad 

30 sending these ^operating system vosoroaods" to die CMWs of the tnhsr panlcjpatus. 
rather ihan continuously exchaagtrtg bitmaps, as would be the. case, with traditional 
"remote, control" products. 

As illustrated sn Figure 28. the original unchanged snapshot is stored its 
a first bitmap 2;0a. A second butnap 210b stores the cosnbinariott of the e-rigssrai 
3S snapshot and any aaaotatioas. Thtts, *hen desired (e.g. , by clkkiog oa a CLEAR button 
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looted «) each participant's Share window, as illustrated in Fig-jte. 2B). -he original 
unchanged snapshot can he restored (I.e.. erasing all afiaotahe-ns} using bitmap 210a . 
Selective erasures can be accomplished by copying isto (i.e. . restoring) the desired 
erased ascs of bitmap 21 Ob with the corresponding portion from bismap C: 10a. 
S Rather than casing a new Shan? window to be created whenever a 

snapshot is. shared, it is. possible to replace the- contents of an existing Shate window vvkh 
a new image. This can be achieved is either of two ways- Firs;, the user can click oa 
the GRAB burton arsd then select a new window whose contents should replace the 
contents of the exssiing Share window. Second, the user can click on the REGRAB 

10 bunofi 5o cause a (presumably modified) session oi the original source window so replace 
the contents of the existing Share wisdow. This is particularly useful when one 
participant desires to share a lesg document that caat:or be ih'sph-yed on (he screen m lis 
entirety. For example, she user might display the fast page of a spreadsheet on his 
screen, use the SHARE button to share that page, discuss and perhaps annotate, ii, thes 

15 return to the spreadsheet application to position to the next page, use ;he REGRAB 

burton to share the new page, and so on. This mechanism tepresstus a simple, effective 
step towsjd application scaring'. 

Further, instead of sharing « snapshot of data on his current screes, a 
user may instead choose ■<:• share a snapshot that had previously fceeti saved as a tile 

20 This is. achieved via the LOAD bason, which causes a dialog box so appear, proispsiag 
■he user to select a file Conversely, via (he SAVE burton, any snapshot may he saves, 
with ail current asnotauons. 

The capabilities described above were carefully selected to he 
psmcui3.f!y effective m envixorirnents where the principal goal is to share existing 

25 information, rather than to crea-e sew information, hi particular, user interfaces are 
designed to make snapshot capture, telepointing and annoratiem extremely easy to use, 
Nevertheless, it is also to b« understood that, instead of sharing snapshots, a blank 
"whiteboard"" ca& also he shared (via rise WHITEBOARD button provided by the 
Rolodex,, Collaboration Initiator, arid active call windows), and that more complex 

30 pairabox capabilities could easily be zdfed for application areas that require such 
capabilities. 

As pointed out previously herein, important teamtes of das present 
system teside is the mannet In which the capabilities and .advantages »f rrmttimecihi -nail 
(MMM), multimedia conference recording (MM'CR;, and multimedia document 
35 management (MMDM) are tightly iategtated with atidlo/video/dsta teleconferencing so 


■ 39- 

provide a ruuki-sedia collaboration system tha? faciutaKs ar> unusutdly higher kvei q{ 
coirjntisiica-ion and collaboration betveea geographically dispersed users tiun lus 
heretofore been achievable by knows prior art systems. Figure 29 is 3 schemaiit: aud 
diagrammatic vis* illustrating how multimedia calls/conferences, MMCR. MMM aod 

5 MMDM work together so provide the above -described features. MM Editing Utilities 
shows suppleoieoting MMM and MMDM may be identic*] 

Haviug already described various examples of aadjo/vwleo/data 
tsHeconfcrersctBg, acxr to b« cc-fc«d«red are various ways of KKcgratxog MMCR. MMM 
and MMDM wish atidiaA-ideoMaia telecouferenelug. For d:is purpose .. basse preferred 

SO approaches; atrd features of each will be considered -Jong with preferred associated 
hardware, and software. 


MinUTSMEDiA DOCUMENTS 

in out axrangeroea-., the creation, storage, retrieval and editing of 
multimedia documents serve as the. basic element cosmos to MMCR, MMM and 

15 M?vlDM. Accordingly., die system advantageously provides, .a aisivarsai S«t« for 
multimedia documents Hiss format defines multimedia doctirt-enrs 35 a collection of 
individual component:; in multiple media combined *ith as overall structure arid siming 
compoi-esf rhst captures the idea-ines. detailed depesdescres. references to, and 
relationships atuosg Use various other cornponcnts. The inibrmaiion provided by -his 

20 structural component forms -he oasis tot spanai layout order of presentation, 
hyaerii&ks, temporal synchroimatksr-,. ere., with respect to ihs eomposmoa of a 
multimedia document. Figure 30 shows the structure of such documents as wsK as their 
relationship wjrh editing a.t:d s-orage. facilities. 

Each of the. components of a moltitnedia document tsses sis own editors 

25 for creating, editing, a«d viewing. In addiiios, each composes^ snay use dedicated 

storage facilities. Multlmsella documents are advat;iagsonsiy structured for authoring, 
storage, playback axsd editing by storing some data, under conventional tils systems and 
some data m special-purpose storage servers as wxU be discussed later. The 
Conventional File System 504 can be used to store ail son •■trne-sensutve. portions of a 

10 multimedia document, is particui&r, the foliowiag ars examples of noa-tL'se-sensiiive 
data that can be stored m a conventional type of computer flic systetn: 


structured sisd unstructured text 


2. raster images 

3- structured grsphsts and sector graphics {e.g., PostScript) 

refereaces to flics is other file systems (video, hi-fidenty etc.) v ;a 
pointers 

S 5. j-e&ijricied Jonas of executable* 

* structure and usving ififotmitiori for all of she above (%pm»l fcyew. 

order of preservation, hyperlinks, temporal synchro-t^auon. etc) 

Of particular importance in mmuroedia documents is support tor 
utse-seasisive n-edsa asd media -has. have synchronisation reqidremecis mih other medsa 

iO components. Some of these time-sensitive media can b-e stored o-i conventional file 
systems while omers vasty require sr^seial-ptitpose storage faculties. 

Examples of rime-seasitive media -ha): can be stored on conventional aie 
systems are .stnaii audio files asd short or tow- quality video clips (e.g. as mighi be 
produced using QuickTime or Video tot Windows). Other examples tnciade window 

15 event lists as supported by ms Window- Event Becord and Play system 512 showa m 
Figure 30. This composes** allows for storing and replaying a users saseraenous with 
application programs by capturing tits requests and events exchanged be-wsen the ciit-.tr-: 
program ano die window system in a time-stamped sequence. A tree this "record" phase, 
(he resulting information is stored m a conventional file that can iate; be retr-eved a-ui 

20 "played" back. During playback me same sequence of window system toques and 

events reoccurs with die same rda-ive inning as when they were recorded, in priot-an 
systems, this capability has been used for creating automated demonstrations in the 
present system it can be used, for example, to reproduce annotated snapshots as they 
occurred ai recewdiag 

23 As described above irs cormecuorj with collaborative workstation 

sofrwsre, St-apshot Share 518 shown la Figure 30 is s utility used in n-uldrneOia calls and 
coirferencit-g fot capturing window or screes snapshots, sharw-g with one or more cat! or 
oc>trierence participants, and permitting group nnnoiaticm, telspointing . and re-grabs. 
Here, -his utility adapted so thai its captured usages and window events can be 

30 recorded by the Window-Event Keeotd and Play system 512 while being used by only 
one person. By synchronising events associated with a video or audio stream to specific 
frame numbers or time ctxles, a multimedia call or conference can be recorded and 
reproduced itt its entirety. Sirmlarly, the same mnctiotsahty is preferably used 10 create 
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smslwnedia au« whose sushoriag *sps are v«tu*«y idesjicaJ to par?«cipa«Ojg m a 
multimedia caU or coofertsr.cc Ofcc-Bgh otbw forms of M MM are sjoi preceded) 

Sortie uaje-seniiiive mcda require dedicated storage setvers m order to 
satisfy retime requsrertterus. Highway a^o/vuieo segment, for example. icqutrfi 
dcdK.sr.ed real-tifxte wdraNideo storage servers, A prefer warn?* of such * «rvcr 
*»« be described fc»r. Nan « be cohered » how the current system suacaurees 
synchrotiizatiot- between liferent racdia composers 

MEDIA SYNCHRONIZATION 

A preferred Banner for providing maitimcda synchronizes will next 
be considered Only multimedia documents with rail-omc material need include 
synchrotron taioas a* sUtanuka. Sywbaantion for Kb situation m*y be 
pxovuSed as described bdow. 

Audio or video segments cass exist withou! being accompanied by ike 
other If AadiQ Mi video are recorded Mnasltarwously ("co-recorded" j. the syssert: 
3lkws ^ C3Sft ***** stor "***««■ *« recorded &ad pUyed back wits automatic 
jynchromzautjR - as woaid tesuir from coav-jyicsai VCRs. laserdiiks, or ttase-divmoa 
multiplexed ("inierfcavcd-} aadSo/video areams. This excludes the aeed to tightly 
synchronise {; « , "lip-syix^separaie audio a;td video sequences. Rather, rehancc <s, on 
iht co-Ktcording zsp&bthy of the Seal-Tixne Aad&Viaeo Storage Server $02 to deliver 
sii closely syochnxfeKl audio and video dlrecdy at its sigaai outputs. 

Each recorded video «eq«t ace is tagged with aaw codes (e.g. SMFFH at 
;/:K> second btsrviU) ot video frame numbm. Bach recorded audio sequeoce « tagged 
with tisae codes (e.g. , SMPTE or MIDI) or, if to-«a:o«Jed «itb video, video Auk 

fMffitiftJS . 

Hie sysreiij «iso provides syrschfosfiiatioa betweea window events and 
audio arid/or video areata*. The foitowng tern* are supported: 

1 ^^^^mx^^^n^dau: lynttoowafan of window 

«"«efi!S to as sud,;o, video, or audio/video stream, using the 
ie&£~ujse media as the ihsmg source. 

2 ■ ■ 

a. synchronization of window svetus to the svstera clock 
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b. syschrcattaiios of the stsxt of ?.a audio, video, or 
audio/video segpcssns so ;he system dock 


If so audio or video is invoked, mdiirx-n^e-dnver, synchrcasjarion \$ 
used throughoy; ihs docatnem. Whenever audio and/or vtdeo ts piay^ng. 
S tnedsa-!sme-syiiCiircm!2AiiCii is used. The system support ■xatsssiioi) bciweert 

fiwch-sae-iiine aad raetka- time syaehroaizatioa whenever a-i ;iudio/vldeo segment is 
ttax&d or siopped.. 

As aa example, viewing ratsj-amcdia document tnight proceed as 

follows; 


syachroai&atiors) 

Nest, surt audio oniy -a "voice snnocaueii") as texs and graphical 
a:ux&&»o&s =>« the share cosuaye (audio «■ tlraisg source for -A-tadow 


Asdio ends, bus sni)oiatjo«s continue (Kwchine-tifiie-drivea 
syndKoaxzsttOti). 

Nest, sssxt coteeonsed atscho/vscko comlnma* ^uh former smuiAOons 

on sat?;? share (audio is timing source for «?irtdow evert!*}. 

Next, s5ax» a new share dariag the cotitiauiag aadioMdeo recording: 

Si-notaaoas happea or both shares {audio is tuning source for wadow 

events), 

A«tho/vidctj stops, asnotatioixs or boiti shares conutK.se 
(rt^chjse-ttise-drivea sytjehrota^atiofi}. 
Document eocss. 
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AUDIO/VIDEO STORAGE 
As described stsove, the present syssfttn can i-xkce nsany 
xpeci;-! -purpose servers & 3 i provide storage of time -sensitive med>a fe §. audio/v-.dec- 
ssteams} and support coordjaaisca with other media This section describes rbe preferred 
arrangement tot authc/vjdeo s;orag« And recording services. 

Although ho; Age sad recording services could fee provided as eacn 
CMW, it is preferable to employ a centralized server 502 coupted to MLAN 10. as 
illustrated in Figure 35 A centralized server as shewn in Figure 31, provides -he 
foito wi Kg «a vantages ' 

1 ■ fcsai assosjtt; of storage hardware required can be far less ■ due to 

better ittil^auon resulting from statistical avetagrug.-. 

2 Bulky and expensive corrsp?i;ss.i;>a ; 'decorriprc'5S!o;i jjardware can be 
pooled or she storage serven and shared Ov roditpie dienus. As a 
result, fewer costpressioa'decornpression engines of higher petfenust-ce 
are retired than sf each workstanoii were equipped with is own 
ompressiors/decoaipress J06 hardware . 

5 - more costJy centralized codexs cast be used to rraaster tnail wide 

area amoag campuses at far lower cojis trsa; attempting f© use data 
WAN technologies . 

4 Fue system administration t'e.g backups and file system replication, 

etc.; are far less costly and higher performance. 

The Real-Time Audio/Video Storage Server SOI shows in Figure 3; A 
structures and manages the audio/video fries recorded and stored on its storage devices. 
Storage devices may typically include computer-controlled VCfo, as wei; as rewritable 
magnetic or optica} disks. For example, serves 503 b Figure 3; A includes disks 60e for 
recording asd playback. Aiiaiog isibrmaiiea is transferred bsrwees disks SOe and the 
A/V Switching Circuitry 30 vis analog VO 63. Control is provided by coattoi S4 
coupled to Data LAN hub "25. 

At a high level, the eentralik-ed aadso/video storage «nd playback server 
502 in Figure .11 A performs the following functions; 


h provides mechanisms for creating, asking, time-stamping . storing, 
remevi-jg, copying, ddetrr-g. and piayntg back some or ail portion;; of at; a-idio/video 

file... 

File Transfer and Replication 

The audio/video file server supports replication of files ott different 
disks managed by the same Ills server to facilitate simaJtaseous access to the sarae files. 
Moreover, rile transfer facilities are provided to support transmission of suOio/v-dco files 
between itself and other audio/video storage and playback engines. Pste transfer can also 
be achieved by using the underlying attdb.'vjdeo network rachitics: servers establish a 
real-time «>dio7video network cosiaccuot; between sheasseivcs .so cse. server can "«Uy 
back" a file while she second server simoit&rieoiisly records it. 

Disk Management 

The storage facilities support specific disk allocation, garbage collection 
and dc fragmentation facEHsies, They also support mapping disks with other disks (for 
tephcatton and staging modes, as appropriate) and rrutpp;r<« disks, vta 3/0 equipment, 
wkh the apprcpnace Video.' Audio rierwosk port. 

Synchro/station support 

Synchronisation between asdio arid video Is ensured by the multiplexing 
scheme nsc-d by the storage snedsa, typically by interleaving the audio and video streams 
in a tiH-e-iiivisxoa-aiuUjptexed fashion Further, if synchronisation is required with other 
stored stedsa (such as window system graphics), then frame numbers. um« cades, or 
other ttixtuig e vests are generated by the storage server. An advantageous way of 
providing this synchronisation is to synchronise record and playback to received frame 
number or tirae-ccxte events. 

Searc/iMg 

To sttppon iatra-ftie searching, at least start.. s>«p, pause, fast forward, 
reverse, and fast reverse eperasjesss are provided. To sapport inter-file searching, 
audio/vkfco tagging, or more generalized "go-to" operations arid rac e hams ms, ssjch as 
frame nutsibers or time code, are snpporfe.d ;-t a seareh- fraction level. 
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Connection Management 

The serve* haudles requests for audso/vjdeo network connections from 
diet-, programs (suci! is. video viewers »ad editors rcsnmng on client workstations;! for 
resftitjie recording and realtime playback of audso/vidso fu'e$. 

Nest to be considered is ho* centralised audio/video storage servets 
provide for re»i-tirae recording and playback of video streams. 

Real-Tims Bisk Delivery 
To .support reaNime audio;' vmeo recording and playback, the storage 
server needs to ptovsje a real-time transmission pa-;h beweea the storage medium and 
ihe appropriate aadkWKseo nerwotk pott for each siatuisioeoiss. elk-m accessing the 
server. For example, if oae uset is vu-.wing 3 video fits at the same time several other 
people are creating and storing r,tw video files or; the same disk, multiple simultaneous 
pa&s 10 the ssorage media are -squired. Similarly, video man* sen; co large distribvuton. 
groups, video databases, and iirjidar inactions may also require simultaneous access ;o 
the same video files, again iisposbg multiple access remsiremer.ss o& the video storage 
capabilities.'. 

For storage servers mat 3.re based on co;np«;er-coniroUed V'CRs or 
rewnsable iaserdisks, a real isme rxansraissiot! path is readily available through U-s d-.recs 
analog connecuoa between s.he. disk or tape and the rework port. However. because of 
ihsj single direct connection, each VCR or Userdisk esc only be accessed by one client 
program ar Use uttse time {multiread laserdsxks are as, exception). Therefore, stouge 
servers based 00 YCRs nad laserdrsks are difficult to scale for multiple recess usage. 
Multiple access to the same maierial is provided by Ok replication and stagmg, winch 
greatiy increases storage requirements and tire nesd for rrs»v 55 jg toformstio-s oukkly 
among storage media units serving dirYeremt users 

Video systems, based on magnetic disks are mote readtiy scalable for 
Simultaneous use by tmdtiple people. A generalised hardware impk-mentauon of such a 
scalable storage aod playback system 502 is ffituttittd in Figure \>T. liidividasl I/O cards 
530 supporting digital and analog 3/0 axe. linked by irms-emrssis digital networking (e.g. 
buses) for file transfer within chassis $32 holding some number of these cards. Multiple 
chassis S.Yi arc baked by inter-chassis ncrwotfcisg. The Digital Video Storage System 
3v;ju;jb!e trotn Parallax Graphics is an example of such a system implementation. 

The baadwidih available for the transfer of fries among disks is 
nlnma-eiy limited by the bandwidth of these iatra-chassis and inter chassis networking 


• 46 • 

For systems that nxe sufficiently powerful video compression schemes, resl-ume delivery 
requirements for 3 small number of users zssi be met by emtsr-jt file system software 
{such as ;he Umx file system), provided that die block -sl/re of the stors.se system ;S 
optimized for video storage and thai sufficient buffering is provided b> she operates 
system software to guarantee continuous flow of she audio/video d«<a. 

Special ■purpose software/hardware solutions car; be provided to 
guarantee higher performar.es under heavier usage or higher bandwidth cortdinons. For 
example, a higher throughput versum of Figure 33 is iliostrased in Figure 35. wrncb uses 
crosspouit swtehmg, such as provided by SCSI Crossbar 540, which increases -he total 
bandwidth of the inter-chassis and imra-ciiassis network, thereby increase she nutnb-er 
of possible sunnt-atseous fsle transfers 

Time N$t*&rk Delivery 
By using the same audio/video format as used for atseUo'vtdeo 
ieiecon.fere.n.e.ing, the audio/video storage system tan leverage «ss previously described 
network fscthues: the MLANs 1.0 c&ti be used to establish a multimedia, network 
cot-needon between client workstations and the audio'video storage servers. 
Audio/Video editors and viewers running ou the client workstation use the same software 
interfaces as the multimedia teleconferencing system to establish these network 

Trie resiiltiag architecture is shows *n Fig-ire 3 IB. Cliem workstations 
use the existing audio/' video network to conrtect to she storage, server's seswork pons. 
These network ports are conr-scted to compression/decompression engines thst plug into 
the server bus These engines compress the sudjo/visieo streams that corns in over the 
network arid store -hem on the local disk, Sinuiarlv, for playback. Use server reads 
stored video segments from its local disk and routes them through the decompression 
engines hack so chest workstations for loos.! display. 

The present system allows for alternative deliver}' strategies, For 
example, some compression algorithms are asymmetric, rnearxmg ^decompression 
require;; tnoch less compute power than compressioa. frs some cases. real-hate 
decompression can evert be done »s software, without requiring any speeiai-psrrpose 
decompression hardware. As a result, there is no need to decompress stored amdso artd 
vjdeo on the storage server arid play it back. st- realtime over the network. Instead, st can 
be more efficient so transfer an entire audjo/vude.0 hie. from the storage server to the 
client workstation, cache it on the workstation's disk, and play is back locally. These 
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observa-ioas lead to a modified architecture as presumed ia Fsgure 3iC. In this 
architecture, clients ^teract with the storage server as foHows: 


* To record video, ciienu acf ap real-tirac audio/video network 

coat-ectiorss to the storage setver a* before (this connection ;:ou)d make 

user of art analog btss). 
" ft) <"«spo : ase to a cottaeaiea request, rise storage server allocates a 

cotaprsssion module to the new ciisru 
o As sooa as the cheat stares recording, the storage setter routes the 

output from, fee compression hardware to an audio/ video file aiioca;ed 

on its. iocai storage devices 
° ?<;r Payback. Oils audio/Video ftse gets transferred over the daia 

network to use client workstation and pre-siagsd on -he workstation's 

focal disk. 

o The ciiea; uses tocai dseoKctptessiott software and/or t:;-rdware to play- 

back the aadio/vscieo or* is to; audio aad video hardware. 

This approach frees up auiito/vsdso network pom and 
compresstoa/decoatpressba engines ok the server. As a fcs server is scaled to 
support, a higher timber of .•ijitsuiuaeoijs reeotdirsg sessions, thereby forther reducing the 
cost of the system. Now that such as architecture car, be employed Tor reasons other 
than coiRpresstojs/deeonrspjifsssoa asyaunetry (such as she ecoaoiaies of the technology of 
the day, existing embedded base in &e eaterprise. etc.) 

MULTIMEDIA CONFERENCE RECORDLNC 
Multimedia conference recording (MMCR) wili next be considered. For 
fali-fearure multimedia deskrop calls and conferencing (e.g. aadioMdeo calls or 
conferences with snapshot share), recording (storage) capabilities are prefets&y provided 
for audio arid video of ai; parties, and also for all shared windows, Wit-sing any 
telepohaing &sd at-aoiauons provided dunrsg the teiccooferstice. Using the trjvJtixftedia 
lyacbroBtatioa fecilisxes described above, these capabilities are provided it! a way such 
that they cart be replayed with accurate correspondence in time to the recorded audio and 
video, stscb as by syachroniztng to frame txu&bm or time cods events. 
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A preferred way of capturing audio and video hots, calls would be so 
record all calls and conferences as if they were tnuUs-party confers&ces s'eveti for 
two-parry calls). using video ssos&tcing, audio ssixuig arsd cut-at-d-pasting. as previously 
described in connection wim Figures- ?-3 i . it wiJi be appreciated tha: MMCR as 
5 described wH advantageously persfct ttset s ;« uheir desktop so re view real-time 

collaboration as it previous;-/- occurred, u-tcludijfsg during a later teleconference. Xhe 
ou-put of 3 MMCR session Is a multimedia document that can be stores, viewed, aad 
edued using the. multimedia document facilities- described e&rSier 

Figure 3 ID sho*$ how ccsfereace recerdi-jg tetates to ihs various 
JO system composiems described earlier. The Multimedia Confeteuee Record 'Flay system 
525 provides she user wish the additions* GUIs {graphical user rriterfacss) and oiher 
functions requited to provide the. previously described MMCR hmcuonahry 

The Coriieteace. Invoke? SiS shown m Figure 31 D is & utility that 
coordinate;, the audio/video calls that must be made to csma*ct the audio/video storage 
15 server 502 with special recording outputs on conference bridge hardware G5 in Figure 
3). The resuiung recording is linked to ^formation identifying the conference, a 
.tutictloti ;-ho performed by this utility 

MULT IMBDIA MAIL 
Now considering multimedia mail (MMM), xi mil be understood ts« 

20 MMM adds to a-e above-described MMCR the capability of delivering delayed 

colbbcratioa, as well as the additional ability to review the information muhrple times 
and. as described hereinafter, to edfs, re-send, and archive it. The captured mforrnsiion 
is preferably a superset of thai caprured during MMCR, eseeps that so other u::er is 
involved ant: the user is gyves s chance to revjew and edit before sending tfeff message. 

25 The Multimedia. Mail system 554 m Figure 3 ID provides the user w;th 

die additional GUIs and other 'unctions required to provide the previously described 
MMM functionality. Multimedia Mad relies oa a conventional Email system 506" shown 
lit F-'jgure 3 ID for creating, transporting, and browsing messages. However, multitnedia 
document editors and v-e^vers are used for creasing and viewing message bodies. 

30 Multimedia documents (as described above) consist of time-insensitive components and 
tune-sensitive cornpeaesjts. The Cos* satio&al Email system SOS relies on the 
Convenoonal File system 504 and Real-Time Ai;dio<-' Video Storage Server 503 for 
storage support. The. tsrne-insensmve components are transported within the 


Coavesiioaaj Emsi! system 506, white the rcai-titne components may be separately 
trasssorted thxoagh the authoMaeo aeswork using fsk transfer utilities Associated with 
the Keal-Time AudioWfeo Storage Serve? 502. 

MULTIMEDIA DOCUMENT MANAGEMENT 
Multimedia document management (MMDM) provides long-!.e.rm t 
higfe-voiuftis storage fax MMCH and MMM. The MMDM system assists m providing 
the following inabilities to a CMW user; 

Maltisftedta docutnetsts can be authored as mail ia the MMM system ©? 
a* c&U/eonferesxe recofdHtgs; is the MMCR system sad. tbea passed oa 
to she MMDM system. 

To she degree supported by eiteraal compatible maJdmsdSa editing aad 
attdsoriag systems, raal&aedis- documents cat* also be authored by- 
mesas other thaa MMM and MMCR. 

Mu:;iraed!3 documents stored wjthin the MMDM system eas be 
reviewed and searched. 

Multimedia d«:atne.nts stored wuhin the MMDM system eas be «se4 as 
tna-enai b the creation of subsequent MMM. 

Mat ttssedia documents stored wltina the MM DM system can be edited 
so create other multimedia docaraenrs. 

The M\iltim«Jia Docameistt Massgeasera system S26 is Figure 3.1 D 

provides the user wah the additional GUIs and other functions tequired to provide the 
previously described MMDM fancdonaiiry The MMDM includes sophisticased 
searching and edhrs-g capabilities in connection wyfe the MMDM. multimedia docurrtent 
such that a user can rapidly access desist selected portions of a stored multimedia 
25 docarftsat. The Specialised Search system 310 in Figutre 30 comprises untitles that allow 
use rs to do smse sophisticated searches across- and within muhimedia documents. This 
includes context- based sad content-based searches (employing operations such as speech 
and image recognition information fihers, etc), time-based searches, and evem-rmed 
searches ;>ixido*' events, tail management events, speech/audio events, etc.) 
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5. 


CLASSES OF COLLABORATION 


The resulting naaMmctiia coikbon-tbts environment achieved by the 


above-described integration of audic/vsc?' 
MM DM iS IShistrated ;n Figure 34. is will 
other users m real-time despise separation:; 


s/dasa selgco&ferensisg, MMCR. MMM asd 
e evident that mch user cas collaborate with, 
f space And time, in addition. coOibcranrsg 
u«« can. access infer-nsuot 1 . already available withis their compuuog 3?sd stifarctutiOfS 
s-ysteros, including info;ma;\on captured from previous collaborators Note its Figure 3<s 
that space, ajtd tirtte separations are supported m the foiJow jag ways-: 


Muhiroedia CiiJs and conference;; 


MMDM access jo stored MMCR and MMM iuforniatio 
MMM directly (i.e., copying -hsiI to ones<eH) 


Collaborative: tsee-tot'ace, ra-ihisnedla document creatton. 


system srsoothiy spans these three v 


nterfaces a nc.svork hmd-jems. dts presets! 


REMOTE ACCESS TO EXPERTISE 

Is order to illustrate how the systesrt i&ay be in-iplsmsxae4 and operated, 
at) exemplary preferred sysseta win be described having features applicable to the 
afcrsmentioned scenario involving remote access to expertise. It is to be understood that 
the syssem may be adapted tor other applications (s\ich as in engineering And 
many factoring; or uses having mors or iess hardware, software &ad operating features 
3«d combined srt various ways. 

Consider the foikswisg scenario invcJvjtig access ftcssa remote sues iv an 
m-fsouse corporate "expers" in the trading of financial jnstnsroetits ;sch as in the 
securities market; 


Tti? focns of the scenario revives around the activities of 3 trader wix; 
5S 3 *J»eci*tat sn securities. The seaing is the start of &» day 4 r aw desk sa a major 
fiaaBciai center {NYC; at a major U.S. >bvc3ukik h&rjk. 

The Expert has been actively watching a particular security over she. past 
week and upon h» arrival into the office, hg aotices ir » on the me. Before going home 
)asi night, he ptev;onsly set up his system to filter overnight news 00 a particular family 
of securities and a yscanty within that fannly. He seaas Use filtered sews and sees & 
story thai may have a ;eng-terra impact on this- secanty sn question. He believes he 
needs -o act now sn ordet to get a good price on -he seetsnry. Also, chtough filtered 
tnaii. he sees &3t his counterpart its Lot-don. who has also bees- watching this security. ;s 
interesred -a getting our Expert's option once he imves at work. 

The £ xpert issues a multimedia taail message on the seenruy ;o the heao. 
of saies worldwide fot use. so worbng with their client base. Also among -he rectp?en« 
>s sr. analyst m the reseatcit department and his counterpart b Loadott. The Expert. 1;; 
preparatba for ht.s previously established "oa-cali" office boars, consults with others 
within the eorpota-jon (using the vKSeocc-afefenciag acts other collaborative techniques 
described above), accesses company records frost his CMW, and analyzes such 
information, employing software -assisted analytic techiuqiKs. Hrs office hours Are now 
st hand, so he srs-ets k saterecnr made, which enables incoming caJls to appear 
automatic:**!)' {vitftour remnrxng the Expert ;o "axtswer his phone" aad elect to accent or 
reject the cail). 

The Expert's computer beeps, indicating an incoming call, aM the 
««3ge of a field representative 201 and his client 202 who are locased at a bank branch 
son-e^here in the U.S. appears in video window 203 of the Expert's screen (shows in 
Fig. 35}. Note that, unless the call is convened to a "■conference*' call (whether 
explicitly via a menu seieettca or implicitly by csJSiag two or snore other participants or 
adding a third pameipaai to a call), the callers wiB see only each other ia the video 
window and will not see then-selves as pan of a video mosaic 

Also illustrated on the Expert's scree;- in 35 ts the Collaboration 
Initiator window 204 from which the Expert can (urging Collaboration Initiator 
software modtiie 161 shown in Fig. 2-0) initiate and control various collaborative 
sessions. Fot example, the user cat; initiate with a selected participant a video caii 
(CALL button) or the addition of that selected participant to an existing video csU (ADD 
button), 3s well as a share session (SHARE button} using a selected window o; region on 
;he screen for a blank tegion via the WHITEBOARD batten for subsequent annotation). 
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The user ear- also invoke his MAIL software {MAIL hurtoxs) and prepare: outgoing or 
cheel; i-icoxm-jg Email messages (she presence of wfcsch is itidkaied by a picture of an 
envelope ti: the dog's ttiotith hi In Box icon 205), as woi! as check for v l called" 
tnessages from other callers (MESSAGES burton) left via she LEAVE WORD button its 
3 video *indo* 203, Video wsndoiv 203 also contaiss buttons Irotrt which marry c> tbese 
atxl certain additional features can be invoked, suet; as hanging up a video call 
(HANGUP button), putting a cadi on hold (HOLD button) . resuming a caii previously p\u 
on hoid fRESUME burton) or routing the audio portion of a call (MUTE bit-ton;. In 
addition, tbe user can sr-voke the recording of a co&fetence by the coij&xence RECORD 

10 burton Also present on the Expert's screes; is a standard desktop wixidow 206 contain^! 
icons (rots winch o-her programs can be ta-mened. 

Rerurning to she example. The Expet? ;S sow engaged in » 
v-irle-xosference wkfc field representative 20; and his ciierrt 202 hi the course of s.hrs 
viiSeoconference, as iiiasrraied is Fig. 36, the field represen;a.iivs shares with she Expert 

15 3 graphical image 210 (pie chart of alien; portfolio holdings • of bis chest's portfoho 
holdings (by clicking on his SHARE husoes, corresponding so the SH ARE bunon in 
virteo window 2QZ of the Expert's screen, axsd selecting thai trsage from his screen, 
resulting in the jhared unage appearing in the. Share window 2i 1 of the: screen of ait 
participants to the share) and begins to discuss the client's investment dilemma The 

20 fteld representative also invokes a corrmtand to seeredy bring up the client profile on the 
Export's screen 

After considering this s&fcrsiasioa, reviewing die shared portfolio and 
asking clarifying questions, the Expert illustrates his advice by creating (using his own 
modelling software) and sharing a new graphicai image 220 (Fig. 3?) with the field 

25 representative and bis chess. Either party ;o the s.h.are can annotate ih&i image using the 
dtawmg tools 221 (and the TEXT button, which permits typed characters to he 
displayed) provided *i;hin Share window 211, or 'cegrab" a s-odified vgrsios of the 
original image (by using the REGRAB button), or remove all such annotations (by using 
the CLEAR button of Share window 21 1), or "gtab" a sew image b share fby clicking 

30 on she GRAB buttes of Share window 211 aad selecutsg that new image frets the 

screen), in addition, aay participant to a ■shared session can add a new participant by 
selecting that participant from the rolodex or quick-di&i list (as described above for video 
■rails and for data cc&fefetidng; and clicking the ADD button of Share wmdow 211 One 
ean also save the shared itrsage (SAVE button), load a ptevionsiy saved is;age ;o be 

35 shared (LOAD burton), or print an Image (PRINT button). 
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Wbjht discussing the Expert's advice, foia representative 20 f retakes 
annotations 232 to image 2Z0 in order to illustrate his concerns. While responding to che 
concerns of field representative 201, &s Expert hem a beep and recedes * v j SU2 i R00ce 
(N * W Cal1 ^ndc* 223} or his screes (wt visible to the field representative and h>s 
t:is<?nO, ndfcaing the existence of a new Incoming call and identifying the caller Ac thi$ 
pojjB, the Expert can accept the ne* call (ACCEPT burton), refuse the new call 
(REFUSE button, which will result ia a message brag displayed on the carter's screen 
dm 5155 Bx ^ n » anavi^te) or add the nw csiler so the Expert' s exunug 
C3l{ (ADD *»*»)• ta *« ^se, the. Expert elects yer another opooo (not shown) ■■ so 
defet fe call and leave- the caller a standard message that the Expert will call back in. X 
niinur.es {la this case. 5 Kji»«ie>, The Expert the;; elects also to defer his existing call, 
telling che field representative aad his cfeot that he will call tea back ia $ minutes, and 
chen elects to return the irxlnsd deferred call 

It should be noted thai the Expert's act of deferring a call results not 
only io a message bemg sent to she caller, but also m the caller's name (and perhaps 
oiher bforatfoa associated with the call, such as the tirtse >Me call wax deterred or ;s to 
be resumed) being displayed in a lis? 230 (see Fig. 3«) or the Expert's screen from winch 
the call can b* reinltiaced. Moreover, the "state" of she call (e.g., the information hems 
shared) is retained so that I? can be recreated when the call i% rcintuted. Unlike a 
"hoSd" described above). defemtig a call actually breaks the Stoical and physwai 
connections, teqniriag Uut the- esUre call he reinitiated by the Collaboration Icuciator and 
che AVNM as describees above. 

Upon retarrnog to the initial deferred call, site Expert engages in a 
videocoaferftnee mth caJler 231, a research analyst who is located 10 floors up ftosn the 
Expert *>th a complex tpenwo regarding a particular wcuniy. Caller 23 i decides to 
add Lo«doa expert 232 to the vsdeocosfereoc« (via the ADD bu-toa in Collabocauon 
initiator window 204} to provide additional iaforroauoa regarding the factual history of 
the security. Opoa selecting the ADD torn©*., video window 7.03 no* display?, as 
iliuaftated m Fig. 3?. a video mosaic consisting of three assail*? images (instead of a 
swgte large image d^plsymg only caller 23!) ol Che. Expert 233, caller 231 and London 
expert 232, 

During this Yideccc-nfexeace, an urgent PRIORITY request (New Call 
window 234} S s received from the Hxpen's boss (who is engaged in a three-patty 
videoconfcrertce call with two members of the bank's operates department and is 
attempt; rig to add the Expert to ihsa call to answer a quick question*. The Expect pots his 
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three-party videcoortferetice or; hold (merely by cHcVarjg the HOLD buster- tr> video 
wjodow 203) and accepts (via ihe aCC£FT basoa of Ne* Call wiadow 234) the urfem 
caH from hss boss, which results is the Expert herns; added to the boss' three-party 
vsdeoccnfetence caiJ . 

S As illustrated in Fig, 39, video wtndnw 203 is now replaced wish a 

four-person video mosaic representing 5 four-party cofiterextce call consisting of ffce 
&spe«233. his boss ?A\ and the f*<s members 242 and 2*3 of the hat-k's opersdons 
department. The Ex-pert qmcidy answers dse boss' question and. by Peking on the 
RESUME bnt-on {of video wmdow 203) adjacent to the names of the osher participants to 

10 the csiS on hold, simultaneously hangs up on £he conference call with his bo;:-: and 

restates his three-party conference call ir-voiving the secwrhtes issue, as illustrated so 
vtde-0 wirsdow 203 of Fig, 40, 

While that call was on hoU'i, however, analyst 23 1 and torsion expett 
232 were still engaged its a two-way videocortferesise (with a blackened pomoa of the 

15 v-.deo mosaic oa their screetts indicating that the Expert was or-, hold; and had shared and 
annotated a graphical image 2S0 (see aaaoiatioat. 25 J to image 250 of Fig. 40} 
Illustrating cextaiss finattciai concents Once the Experx resumed die cah. analyst 2>S 
idded the Expert to the share session, eausmg Share window 21 i containing annotated 
image 250 to appear on the Expert's screen Optionally, snapshot sharing could progress 

21} while the v«|eo was. on hold. 

Before concluding his conference regarding the securities, the Bxpen 
receives notification of an it-coming mttitsrnedia ma; I message - e.g., s beep 
accompanied by the appearance of as envelope 252 m the dog's mouth ;n in Bos; icon 
2Q5 shov-m in Fig. 40. Once he cot-dudes his call, he qusekiy scans his isicorrijrrg 

25 multimedia mail message by clicking on in Box scon 205, vhicti invokes hts mad 
software, aad the a selecting the. itscotmrtg message for a stuck sears, as generally 
illustrated to the top two wisdoms of Fig. 2B. He decides it cats wait for farther review- 
as she sender is an analyst other than the o«e helping o;s hss s«canry onesdoa. 

He then reiaiUatcs (by selecting deferred caii indicator 230, shows m 

3D Fig 40) his deferred call with field representative 201 sad his dies; 202. as shown it; 
Pig 41. Note that the. Ml state of the call js also recreated, including restoration of 
previously shared image 2.20 with annotations 222 as they existed when the call was 
deferred {see Fig. <7s. Note also io Fig *l that, having reviewed his ordy attread 
jsscormng frmltisneola mail message. In Bos icon 205 no longer shows an et* elope to the 

35 dog's month, tndscasing that the Expert currently has no unread :.r;co-rtlag messages. 
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As the &zpsn continues to provide advice sad pricmg sr-fori-nation to 
field rcptewmtatlve 201, he receive notificauon of three priority calls 26; -263 in short 
accession. Call 26; is ifc Head of Sales for ifce Chicago office. Working a- home, she 
had instructed her CMW to alert her of ah tsrgent news ot messages, and was 
subsequently alerted to the arrival of the Expert's earlier mutamedu mail message . Call 
262 is m urfent tatemaUonal c*!l. Call 263 is ftvm the Head of Sates in Los Angeles. 
The £xpen quickJy winds daws and diets cortciudes his call with field reprs.seTua.iive >0s 

The Expett notes s'rora calf indicator 262 that this call * not only an 
international call (shown sn the top pernors of she New Csii window), but he realizes it is 
trotn a laptop user ;s the field In Centra! Mexico. The Expen elects 10 prioritize his calis 
\s. the following msruter 265, 261 and 263. He therefore quickly answers call 28 i {by 
clicking cti its ACCEPT burton) asd puts Oiac call on hold wfcds deferring call 263 in the 
manner discussed above He rhsrs proceeds so accept ;.he call identified by international 
call tMbcaor 262. 

Note its 42 deferred call -tidicator 271 arid the indicator for the call 
placed on hold faext to -.he highlighted RESUME burtors in video window 203}. as we!! 
as die image of c;;Hsr 272 from die laptop in the field in Central Mexico. Although 
Mexican talter 272 is outdoors and has so direct access so any wired telephone 
connection, tiss laptop has two wireless modetns pennirting dial-up access to two data 
connections in the scares: field office (through which h;s calis were rented). The system; 
aototnaocally {based tspos the laptop s registered service, capacities) allocated one 
ccttrtfcaoa for an analog telephone voice call (using hss laptop's built-in microphone and 
speaker and the Expert's compiiteHategrsted telephony capabilities) to provide audio 
teleconferencing. The othet cotsneerion provides coitsroi, data coaferenciag and one- way 
digital video (i.e., the laptop user cannot sec. she image of the Expert) from the laptop's 
baiit-m casnera, alt;e;t as a vs?y slow frame rate (e.g.. 3-10 srnail frames per second) d«e 
to the. relatively stow dial-tap phone coaneeuoti. 

It is hnportant to nose slat, despite the limited capabilities of ;.he wireless 
laptop equipment, -he present system tjceertunodates such capabilities, suppktnenting an 
audio telephone correction with linked (i.e., relatively slow) one- way video and data 
conferesjcirtg rurtaiosaiiry. As telephony and video compression rechitoiogjes improve, 
the present system win accommodate snch improvements automatically. Moreover, even 
with one participant ro a telficon/ereace having limited capabilities, other participants 
need no: be reduced to this "lowest cormrtos denominator" For example, additional 
participants could be added so site call dlusttated in fig. 42 as described above, ants such 
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participants could have full videoconferencing, data confere-st sng and othec collaborative 
funcsioaahty vis-a-vss one aaofhet. while having ij -Kited fuscstonaiity only with easier 
272. 

As hts day evohed, Use off-site salesperson 271 in Mexico was notified 
5 &y his manager through she laptop about a issw security and became convinced rest his 
client vL-o-uid favt parocsdaj interest, i.r: dus issue. Is* salesperson therefore' decided to 
contact the Expert as shown in Figure 42. While discussing the security issues, the 
Expert aga-rs shares ail captured graphs, charts, etc. 

The salesperson 272 also needs the Expert's Isesp on ano'Jicr issue. He 
10 has hard copy only of a client's portfolio and needs seres advice on hs composition 
before he tnee-s wish the client somorrow He says he win fax n to ths Expttx tor 
analysis. Upon receiving the fax -oh his CMW, vis compter -rategrated fax-she Expers 
asks if he should either seas ths Mexicats caller :3 "QuickTime" stK-vie u lower quality 
compressed video s-jatidsrd from Apple Computer) on his laptop tonight ot send a 
15 higtset -quality CD v\s F-'edX tomorrow ~ the nodos being mat the Expert can produce ar> 
actual vyteo psesentatson wish models and asrnotations in video form. The salesperson 
car. then pisy it -o his client tomorrow aneroooa and rs wiil be as if the Expert is us the 
room. The Mexican caller decides he womd prefer the CD. 

Conurtuing with sins scenario, the Expert learns, in the course of f-is call 
20 wiiiJ femose laptop caller 2*?2, th&r he missed an iraportaxrr issue during Ins previous 
o,us<cfc M-3T: oi In* moutnmg ou.uv.uieJia maii message. The Expert is upset mas she 
sender of the message did not utilize the "video highlight" feature to bighiighs snis aspect 
ot the rrsessage. This feature permits the composer of she message so define y tags" (e.g., 
by clicking a TAG b«raoa, not shown) during record tires winch are s-ored with the 
23 message along wish a "time stamp", aud which cause a predefined or selectable audio 

and/or visual indicator to be played/displayed as that precise point in tise message during 
playback, 

Becaase dus m«e relaxes to -Jss trailer mat the Expert has on hold, -he 
Expert decsdes so merge sits rwo calls together by adding Use call of; bold to his existing 
30 call. As no-ed above, both me Expen and the previously acid caller wips have full video 
capabilities vis-a-vis one another and will see a three- way mosaic image (with she ires?e 
of caller 2"2 at a slower frame rate), whereas caller 772 will have access only so the 
audio portion of this three-way conference call, -hough he wdi have data conferencing 
functionality with both of the omsx psitirapasts. 
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The Hxperi forwards, tlte rrnrihmedia tsail message so both caller 272 
tfsc nine- paroeipaat, A*sd ail tsree of them review site video enclosure ^ greaser detail 
ami discuss the concern raised by caller ??> They share certain relevant data a* 
described above and realize that they need to ask a quick question Pi another remote 
expen They add Utat expert so the call tiesumng sa the addition of a fourth image to ihc 
video mosaic, also not si-ownl for less stars a msatste white they obtasn a stuck 3cs*er so 
their question. They riser- continue dietr rhree-way call uti-d the Expert prowde? tm 
ad v see asd -.hers adjcutr;-; the call 

The Expert composes a new muiti«:£dsa rcasi message, seeorthrrg his 
image And audio synefcrocized us Ascribed zbovs) to the screen displays resulting ftorr; 
his sjrtiUiiantous interaction wifih hrs CMW {e.g., running a program thai performs 
certain calculations and displays a graph while the Expert IS lustre-set; certain po-nis by 
reiepoimiag oa the screen, duriag which utne his is* age and spokea words are also 
caprured;. He- sends this message to a -iwuber of salesforce tecipiotus whose Ideamiss 
ar;-: detersiased automatically by an outgoing: mail filter that utilizes a database of 
Ini'csrmauojs oa each potential ;ecipleos {e g v selecting ordy these whose eiiert-s have 
investment policies ^inch allow rhls type of investment). 

The Expert mess receives aa audio a-sd vjsuai rerasader {mt shows) ttsst 
a par-usulax video feed {e.g.. & short segment of a financial eable television sco* 
fearing rtew fsnaaaa; inssruments) will oe triggered suromasolly >a a sew munaes. He 
uses this time ?c- search his local seonrks.es database, wttioh is dynamically updated frotss. 
f.auxrai iRfcroutioB feeds (e.g , prepared from a broadcast texmaJ strews of current 
financial events with indexed headers that autortia.tie.ajly applies data flhe-s fo select 
mcomsng events relaung to ccruats securities}. The video feed is then. dis»l«ys?d oa the 
Bspsn's screen and he watches sals short video segment. 

After aaaiyzjag this exstetnely up-to-date iuforruairon, me Expert the« 
tehx-sates his previews*? deferred cail, from indicator 27 i show, tit Fig 42. whtch he 
ici-.ws rs from the Head of Saks ts Los Angeles, who is seekscg to provide hi,, prime 
cherts with sceuris-,es advice ors art-other securities transaction b*sed upon -he most recent 
avsjiable information The Experr^ caJi is sot answered directly, sisoogh he. receives a 
shnn prerecorded video message {kit fey &e caller who had to ieave his home ior s 
meeting across town soon after bis priority mesissge was deferred) asking that the Expetr 
leave htm a rfiuhimedia mail reply message wufa jdvtc* for a pamcubr chem, a«d 
exphsrtirjg tha? he wth access tfess message remotely from his laptop as soorr as his 
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meeting is concluded. The Expert compiles with this request at-d. composes and sends 
this mail message . 

The &x?CJi then receives sr. audio and visual reminder on hu screen 
indicating thss his office hour; will end in two minutes. He switches ftorn "nnercorti" 
3 mode to "telephone" mode so that he will no longer be disturbed without an oppomtnuy 
so teiect i&coniisig calls the Nc*- Call vci&dow described above. He then rece-ves snc 
accepts a fitui call concerning ass issue from ;«s electronic meenng several months ago. 
v*hsch was tecotded in its* entirety. 

The Expert accesses this recorded meeting from his "corpora-e 

10 memory He searches the recorded meeting {which appears in a second video window 
on his screen as would a hvs meeting, along wi-h standard controls for 
siop/play/rewiod/mi forward/etc } for an evtmt thai will trigger his memory using his 
fast forward cotsaois, bat cannot locate dte desired portion of the meeting. He :hen 
elects to search the ASCII text, log {which was automatically extracted sts the bsekgtont-ii 

15 after the meeting had been recorded, using the iatest voice recognition iechniqsjesj, but 
soli caruiot locate the desired ponton of die meeting. Finally, he applies an mlotrnation 
fillet ro perform .i content-orien^d (rather thsn literal) search and fisds the portion of the 
meeting lie was seeking. After qmekiy reviewing this short portion of the previously 
recorded meeting, dse E>;perr responds to the caller's qisescton, adjourn; the cail and 

20 conciisdes hss office hours. 

It shoidd be noted &«£ the above scenario involves ma^y .stare -<;f -tf -e -:»rt 
desktop tools (e.g., vsdeo ar;d infomsatioa feeds, infortnatioa filtering and voice 
recognition) -hat cats be ieverageo by our Expert daring vtdeoc.oafereiK.sng, data 
conferencing and other collaborative activists* provided by the present system - because 

25 this system, i-istead of providing a dedicated videoconferencing system, provides a 
desktop multimedia collaboration system that integrates into the Expert's existing 
workstatsco/LANTWAN esvuomnent. 

It should also be noted that ail of the preceding collaborative activities in 
this scenario took place during a relatively short portion of the expert's day (e g., less 

20 than art hour of etimulstive time) while the Expen remained in bis office and continued to 
utilize the tools and information! available from his desktop. Previously. s.ach a scenario 
would not have been possible; because many of these activities could have takes place 
only wim facc-;o -face collaboration, which in musy circumstances is not feasible or 
economical and which thus may well have resulted in a loss of die associated business 

35 opportunities. 
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Many rsodjacauons asd variauoas cam b« made m hardware, software. 

opera-ion, uses, protocols and data format*. Fot cxampk. for asr-am applicatioBS. =t 
wili b& uxefa} ;<:■ provide some or a\\ of the audio/video sigrjgfs iss digis-s! form. 

The apphcaijon is divided out of sppiicauca No. $410665.5. Sens! No. 
2,282,506 k> whsch reference should fcr ajsde. Attention s$ also drawn to the foHowixsi? 
ipp!icstlo:« likewise divided o«; vl th&t spphcasion: 
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CLAIMS 


A teleconferencing system. comprising 

*» AV pith configured to carry AV skoals Among 3 plurality of wofksutxwt. 

the. AV stgjjsis representing video images arid spoken audio of a corresponding 
plurality of participates; 

an aV conference ataaager configured so manage a videoccfiifetetice rtiirifie, 
which she video uaage and spoken audio of oae of (he participants, can be 
reptoduced by a monitor js communication with -he workstation of another of 
she participants, 

a data conference ai&aagcr configured fo (Manage a data conference during which 
data is shared .among the participants aad reproduced by the -noniton; of their 
respective workstations: and, at least one of 

3 ntirUitaedia snail system configured to store, as a multimedia mail message, a- 
least o.-e of tat; ssgrsais selected from the group eoasistiag of video, audio and 
dasa signals generated 3? the workstation of a prepanag participant, during the 
videoconrereacs and dasa.eonfere.tice, and, capable of foraatding fhs tntjitimedja 
mad ?ae;;::as.e ;o a receiving participant: oj a tasiiiunetha conft-rence recorder 
capitis of recording the video, audio and data signals. 

The telecosiiereucmg system of Claim I, further comprising. 

art AV .storage server capable of stonag AV signals prepared by the nmisunedhi 

•nasi system or recorded by the multimedia conference recorder, wherein 

CO 

the AV signals carried from The workstations to the AV siorage server cat- he 

either analog or digital signals;: 

(si) 

the aV signals earned from tlte AV storage server to the workstations eat; be 

either analog ot digital sig&ab; and 

('lis.) 

AV signals cart be stored ia the AV storage server either its analog or digital 
formats. 


A teleconferencing system, comprising 


(as data conference capture tools eapshie of capping data getsmted a; a 

wcrkststjoa of a preparing parricipa-, 
(b) aanota-iorf tools capable of annotating the captored data, sad 
(t? a mo-iinvsdia resit system configured to prepare and stote, as a tn-jitimctha rrad 
fSiSRSsge, the espmred and annotated data, and configured :c forward the 
GiulutJietha mail message. i# a receiving partKjpaat. such that -he .-nukiroccha 
toa-l message can be received at 3 venue selected from shs group cosseting ol. 
<*> 

in rea; time at a location tetaoved from the preparing participant, 
til) 

at 4 different ttase at the saae location as die roe^age was prepared; and 
(in) 

at a different time at a Jccatica removed from -he preparing participant 

Ihe tekeonfecencmg system of claim 3. -vheresr; the miiiiira«d;a mail system ;S :n 
cutamuakatjoa With a graphical atataauoas originator device uh;ch cart gerterate. <uv* 
and replay animated graphical images uj be included in she muinmedia mail message 

5 Ttte ielKonffrtacing system of claim 4, farther including AV capture and 
reptodoction capabilities ior eapmnog atrf reproducing the video images and spoken 
atsdto of the paxttcipaats. 

6. lit? teieeorfcescing system of daun. 3, whersva 'he captured video images and 
aadjct includes video images and spokes audio directly associated with the preparing 
participant arid at leas; on? other video image and spoken audio no? directly associated 
with the preparing participant, 

V\ The teisconferescisg sys&s* of daint 5. farther conaprisiag: 
<a> a message marker capable <jf defining 3 srtarksd portion of the ntulrtnardja msii 
message, 

wherein tfts marked pomes eaa be selectively displayed by the receding participant 
when the multimedia mail message is reproduced 


B. lh& Ts!ecor*f«rsaciag system of claim 7. funher comprising a tag searcher 
configured to search for a defined tag, such that the pott-osi of she message can be 
&eS«ctiveiy displayed. 

9 The teleconferencing system of ciairrs 8, wherein the tag searcher can search a 
pUrrahty of mnitunedia trtaii s-essages -o locate the defined tag . 

10. The ieI«:o:)fetet:C(!)g system of claim 9, further comprising: 

fa} data conference manager configured to raanage a data conference curing which 

data is shared among a plurality of she participants and displayed on mormon 

iaciuded a; their respective workstations: 
<;b.j a multimedia conference recorder eapabie- of synchronising and recording hoth 

the video image and spoken audio of the participants during she yideocotvfcrence 

and the data shared dunng the data cemfeje-jee 

1 1 . The seiseortferetoxig system of cimm 10. further comprising: 

;» a multimedia mail depository associated with fire- receiving participant and being 
operable to teceive and score multimedia mail messages under directum of The 
preparing: panicipam, 

12. The teleconferencing system of ctstra i J, wherein the mad depository ;$ 
operable;, by the preparing participam . so receive mad messages haviag 3t teas; one 
componem selected from the group coasistrog of audio, video and data components. 

J 3. A teleconferencing system for conducting a Teleconference among a pimahsy of 
participants having workstations with associated monitors for displaying visual images, 
and wuh associated AV capture and reproduction capabilities for capturing and 
reproducing video images arid spokes audio of the participants, the workstations being 
interconnected by a first network, the network providing a data path for carrying digital 
data signals among the workstation*, the teieconfereocisig system comprising; 
(a) an AV conference manager for managing a videocenfetence during which she 
video image and spoken audio of :i firs; of the participants- is captured at the 
workstation of the first participant axtd reproduced at the workstation of a second 
of she participants; m& 
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fb> a fiiu!t!st:edia mad -sy^en; for p!t?parir.g and storing, ,ss a multimedia mas; 
tnessage, ihe vsdeo tsnage axsi apoken audio generated and captured at ass 
workstation of a prepssing partidp&ai, and for forwarding she tsyitimedia mad 
menage to j receiving panicjp&ss and reproducing the captured vjd^o trj-age and 
spoken, audio of the preparing parhesp.-srs; a? Use worksta&oti of the receives; 
participant, whereby {he AV coherence maruges and tridti-twdss mad ^srem ? 
ihe awooased AV capture cmd reproduction e.ap<;foun\e:: 

54 A ieurixssjfcrencifig system for conducting a teleconference arnorig a pJuraiov of 
participate havwg works^dor>s wsth associated tno-usctrs. tor display^ vsstis; itnage*. 
and *kts associated AV capture and reproduction capabditics for capturing ;i,«d 
{cptc-diictng vsdeo images and >:pokers audio of die participants, die workstations bests? 
isiercv^cted by s fins network, the network provsdsag a data path for carrying digstai 

aata sigmii aosong the workstations, the teleconferencing system ccmpri&ng- 
<3) as AV eof-forsnee manager for managiag a Mdeoeorsfereace during whic;: the 

v;dco * triage and spokes; audio c<? one of the participants can be sentenced at the 

worksrauor; ot another of che participants; 
C»! a rrsai-tntedta mail system for preparing and storing, at a frsuitiraedi,: -rum 

message, the video image and spoken audio generated as the workstation of a 

preparing psrt-cspasu, sad for retrieving the •rsuttttriedia mad message for 

forwarding to a receiving participant; 
it.? a r&aliimeaja coaferecce recotder for recorcsrsg the vuieo image and spoken 

audio of the participants durreg the videocessfetettce, and 
td) S3 AV fiie sys-errs for storing and retnevia^ both ;he video image and spoken 

audio of die preparing participant and the recorded video image assd spoken 

audso. 


IS. a teleconferencing system fot conducting a teleconference among a plurality of 
pruespaats having ^orkstadons with associated tnoanots for displaying visual images, 
and associated. AV capture asd reptcsdiicoo?) capabilities for capturing and 
reproducing video images. a.nd spokes audio of said participants, said workstations being 
interconnected by a first network, said network providing a data path for carrying digital 
data s-jgrais amotsg sajd wetksutrons. the tesetotifsnsntritig system coiftprntng 
faj an AV pst h for tarryvag AV sigtsais among said ft-oikstatsons, .tasd AV sig-sais 
represet-fi-ig vufeo jnjages, -snd'or spoken atsdsc of «id panscipants. 
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(h) an AV conference nranager for ttttosgitig a viiSeoconfersHce during which the 
viaco image and spoken audio of one of said participants cars fee reproduced it the 
workiiatiotj of asoiiiEr of said partsapants; 

(c:j a isuUstsedia mail system for storing, as a multimedia mail sjutssa^e, AV signals 
.1 generated si the workstation of a preparing participant, and for forwarding said 
ixtttiusnedia mail rrse.s5.age to a receding participant; and 

(d) a tntnttmedia conference recorder for reeortStsg the AV signals representing the 
video images and spoken a-idks of said participants daring said vidsocotifsrsrice. 
whereby said AV path carnes the AV signals gesierated during sa*d videoconfetence, 
iO recorded by sates multimedia conference recorder, and secluded in said rcuhiraedia maii 
message. 


i6. A teleconferencing system for conducting a teleconference among a plurality of 
participants having workstations with asitsxated mooitots for displaying visual images, 
and with associated AV capture and reproducii-xi capabilities i'or capturing a.r ; d 

!.S reproducing video images and spoke;; a«d-o of ss«3 participants, said workstations being 
interconnected by a first racswork, said network providing a data path tor carrying dsgitai 
data signals among said workstations, the teieconfcrsnciug system coinprismg: 
(a) a data conference raaxtager for .'managing a data conference during which data 
are shared among a plurality of said participants arid displayed on me monitors of their 

20 respective workstations, said data conference controller including 
(i> oapfure tods for capatriag m4 data to be shared, arid 
Mi) annotation tools tor artnotatmg said capered data; and 
!» a multimedia mail system for preparing and storing, as a ratihanedia mail 
message, data generated at the workstation of a preparing participant, and for forwarding 

25 said multimedia mail message to a receiving participant, whatetiy saiti tmiiiimedia mail 
message is prepared using said capture and annotation toots. 


